Replace NA in all columns with dplyr (#rstats)

Searching various dplyr help pages like those in the RStudio blog did not reveal a dplyr function for converting all NAs across an entire data frame (i.e., all columns / all variables) into a value. My attempts with replace_na and mutate_each failed.

A psychic would have (may have?) heard in my mind, “I just want these NAs to show as blanks so I will not get bombarded with questions about the meaning of NA.”

I was rescued by the base R function

replace

which I suppose really means one or more people in the R Development Core Team rescued me.

# Borrowed from https://blog.rstudio.com/2015/09/13/tidyr-0-3-0/

df <- dplyr::data_frame( year = c(<span class="hljs-number">2015</span>, <span class="hljs-literal">NA</span>, <span class="hljs-literal">NA</span>, <span class="hljs-literal">NA</span>), trt = c(<span class="hljs-string">"A"</span>, <span class="hljs-literal">NA</span>, <span class="hljs-string">"B"</span>, <span class="hljs-literal">NA</span>) )

DF1.JPG

df2 <- df %>% replace(., is.na(.), "")

DF2.png

Advertisements
Replace NA in all columns with dplyr (#rstats)

Lost Internet and login solution – Linux upgrade (Ubuntu 16.04)

Upgrading from Ubuntu 14.04 to 16.04 (after resizing my partitions using GParted) caused my Asus K54C laptop to lose Internet access; boot to the terminal with the usual blinking cursor instead of the usual, attractive login screen; and my login credentials to fail after loading the usual screen with:

 sudo lightdm

I use GRUB to enable the option to boot into Linux or Windows 10 (“dual boot”).

Entering a ping statement revealed the dead Internet connection.

My solution included unknown elements among the following steps:

1. Keying “e” with the Ubuntu option highlighted in GRUB and cursoring down to:

linux /boot/etc...

and adding at the far-right:

acpi=off

Then hitting F10 to boot.

Note: I repeated step 1 at least twice, before and after performing the following steps.

2. Re-install lightdm.

  1. sudo apt-get remove lightdm* --purge
  2. sudo apt-get install lightdm*

(you may choose to perform all the steps under “Took me two days” on this Stackoverflow page)

3. Whatever .Xauthority is, it is associated with many issues with later Ubuntu version so try:

sudo mv ~/.Xauthority ~/old.Xauthority

4. Repair packages with:

sudo dpkg --configure -a

Note: I answered “Y” when prompted about a version concern

5. A couple reboots, including a repeat of step 1 each time.

Somehow, voila! (borrowed this image from ubuntuguide.net)

ubuntulogin.png

Finally, I entered the following after successfully logging-in to the usual interface (lightdm). It apparently cleans-up your configuration.

sudo apt-get autoremove --purge

Lost Internet and login solution – Linux upgrade (Ubuntu 16.04)

Randomized IDs for a patient list in R

The HIPAA privacy rule constrains sharing even the initial three digits of a particular patient’s zip code unless the population is large enough, so we must be careful when sharing patient data.

zip.png

Say you have a patient list stored in data frame sumfrm and you want to assign a random number to each patient.

Here’s one way to do it. I made sure all of the values were unique in the random number list in the very unlikely event that my use of the abs function (absolute value) caused a number to appear more than once.

# Assign sumfrm as mtcars and pretend a car is a patient for example purposes
sumset <- mtcars

# The switch: Either generate the random numbers again or do not (0)
reprand <- 1
# Initial seed value
seedstart <- 1234
# Initial iteration
i <- 1
randdev <- function() {
# Check the switch
while (reprand == 1) {
# Notice the need to set the seed each time so you can deidentify later
set.seed(seedstart + i)
print(paste0("Seed is ", seedstart + i))
randnums <- abs(round(rnorm(nrow(sumfrm))*1000000000))
randnums_unique <- unique(randnums)
randnums_length <- length(randnums)
randnums_unique_length <- length(randnums_unique)
if (randnums_length == randnums_unique_length) {
reprand <- 0
return(data.frame(randnums))
} else {
reprand <- 1
}
print(paste0("Iteration is #", i))
}
}
randnums_out <- randdev()
colnames(randnums_out) <- "Rand Px #"
# Bind the random number column to the original data frame
sumfrm$RandPx_Num <- randnums_out

We see our new random patient number on the right. Incidentally, I hear Mr. Datsun 710 is healing quite well.

rand_cars_2.png

Make sure to record the seed number so you can replace it for:

seedstart + i

The biggest surprise for me was that I have to set the seed every iteration. Random number functions that use a seed algorithmically and consistently change the numbers selected in every time the function is called, until the set is set again.
rand_r.png

Randomized IDs for a patient list in R

MiKTeX install for R (Windows) – PDF output (R markdown)

Some notes after investing hours to generate PDF output in R using Yihui Xie’s knitr capability in RStudio (Knit to PDF).knitr.JPG

I had reached an impasse of forgotten details in my inability to “Knit to PDF” an R Markdown file in R Studio. Continued work on my in-development R package (FBadstats, for analyzing Facebook ads performance) led me to an R Studio Package Development website that included:

“To build manuals and vignettes you’ll also need to install the MikTeX LaTeX distribution for Windows which you can download from here: http://miktex.org/download.”

I learned MiKTeX was usually pronounced MIHK-tek and the latest setup executable was here. The install was facilitated by Søren L Kristiansen‘s very nice tutorial.

However, I kept hitting LaTex Error: File `url.sty` not found. Note: wrapfig.sty is shown in this screenshot.

Error_miktex.JPG

Ultimately, this was my solution. Some of these steps were due to access restrictions on my work laptop that made other manual installation of package instructions fail:

    • In MiKTex Package Manager (should be in the Start menu under your MiKTex folder), click Repository -> Change Package Repository -> “Packages shall be installed from a directory.” then Next -> select your browser’s downloads folder -> Finish
    • Download the missing basename concatenated with “.tar.lzma” (e.g., “url.sty” indicated downloading “url.tar.lzma”) from https://www.ctan.org/tex-archive/systems/win32/miktex/tm/packages and place the downloaded file in the folder of the above step
      [Note: this worked even though I downloaded the 64-bit version of MiKTex and the aforementioned URL included win32]
    • At the command prompt enter the following, changing “url” to the appropriate package name (i.e., the basename in the above step):
      mpm –verbose –install url
    • Retry “Knit to PDF” in RStudio

Eventually, all errors disappeared and I experienced a knit PDF.

knitPDF

MiKTeX install for R (Windows) – PDF output (R markdown)

Bing Maps API for driving distance in R

Adding to my earlier post about finding a workaround for errors encountered while using the Bing Maps API option of the georoute function in the taRifx.geo package:

In specific, this error is what I attribute to the package no longer being updated:

BingFail

ERROR: “Bing Maps API return status code 404 – Not Found”.

This code is hopefully self-explanatory. Make sure to insert your Bing key for the bingkey value after acquiring it from Microsoft [LINK].

library(tidyverse)
library(geosphere)
library(taRifx.geo)
# North Carolina SNAP-accepting retailers
# downloaded from https://www.fns.usda.gov/snap/retailerlocator
snapNC <- read.csv('SNAP_RetailerLocator_NC.csv')
rowct <- nrow(snapNC)
for (k in 1:rowct){
tryCatch({
outfil_snap <- snapNC %>% slice(k)
from_mat <- t(c(outfil_snap$Latitude, outfil_snap$Longitude))
from_mat_in <- apply(from_mat, 1, paste0, collapse=",")
# same values because each location will be mapped to every other one
to_mat <- from_mat
to_mat_in <- from_mat_in
xmat <- rbind(from_mat, to_mat)

distunit <- 'mi'
bingkey <- 'PLACE_YOUR_KEY_HERE_AS_ACQUIRED_FROM_https://www.bingmapsportal.com/'
starturl <- 'http://dev.virtualearth.net/REST/V1/Routes?wp.0='
fromtourl <- str_c(starturl, from_mat_in, '&wp.1=', to_mat_in, '&distanceUnit=', distunit, '&key=', bingkey)
# do not map same location
if (!identical(from_mat, to_mat)){
u1 <- RCurl::getURL(fromtourl)
u2 <- RJSONIO::fromJSON(u1, simplify=FALSE)
outfil_snap$travelDistance <- u2$resourceSets[[1]]$resources[[1]]$travelDistance

if (k == 1){
outfil_snap_dist <- outfil_snap
} else {
outfil_snap_dist <- rbind(outfil_snap_dist, outfil_snap)
}
}
message(str_c("completed k iteration: ", k))
},
warning = function(w) {
print (str_c('Warning on iteration', k))
}, error = function(e) {
print (str_c('Error on iteration', k))
}
)
}

 

Bing Maps API for driving distance in R

Marquel Russell review – online coaching

As I continue to consult with a variety of persons and organizations who are either helping me understand my options for recouping at least some of the thousands I gave business and Internet marketing coach, Marquel Russell, or who are already performing their own investigations that will likely not yield any direct reward to me, I simply want to encourage any prospective client of his, or any coach, to:

    1. Read contracts carefully and ask questions. Do not allows yourself to be rushed into an expensive decision. Think about the implications! Ask: Why is this person rushing me to a decision and / or not communicating details? Does this person understand my personality and particular objectives? Does this person seem like someone who will work hard to help me think through complex challenges?

Why does this contract disallow refunds under any circumstances? Why does it have forced arbitration language? Why is a state’s laws presented in the contract that is not my own or the coach’s? How has this coach achieved their purported great success without any reviewer commenting on qualities that some view negatively?

      1. Trust your intuition.
      2. Reach out to clients not featured on the testimonial pages.
      3. First, deeply explore more affordable options for Internet marketing / branding assistance. As I hopefully made clear in my Matrixx review / “Laugh At My Pain” post, cost does not predict quality in the coaching / personal development space. Rick Mulready’s free podcast, “The Art of Paid Traffic“, is an amazing resource that offers the same ideas you might pay thousands to learn from someone else, augmented by Rick and his guests’ novel discoveries. Someone also suggested exploring the materials of “Private Label Rights“, although as we see so often in this industry, there’s obvious deception. Let’s just say what I see on this page looks very familiar but it costs less than $100.
      4. Approach marketing of any kind as a risky endeavor, and so, while success may come, anyone presenting examples of “rapid success” is likely presenting atypical success. Similarly, a “fail proof system” likely has produced unsatisfying results for many.
      5. If a coach says anything about their status, such as being “world-renowned”, research whether this appears to be accurate. For example, Russell Brunson may be the most famous Internet marketing expert, does a possible “world-renowned” digital marketing coach appear online to be anywhere as popular as Mr. Brunson? Compare a coach’s claims about their status and expertise to their online presence versus others in their field. This is a must! For example, Alexa rankings will give you a hint of the popularity of their website.
      6. Review what the Federal Trade Commission has said about business coaching.
      7. Consider that a coach who deceives might encourage you to do the same. I was advised by Marquel to call myself a “world-renowned” relationship expert. This could have put me at legal risk and damaged my personal reputation. I have no qualifications or professional experiences to justify such a descriptor. Sure, I like helping friends with romantic relationship concerns and would love to earn money to do so, but no one has ever paid me or considered my language that of an authority.
      8. Watch for mixed messages. Does the person say online in some places that he / she cannot help you get rick quickly but in other places say he helps clients quickly earn a lot of money? Does the speak to acting ethically and yet your intuition is that ethics are not of high-value to the person?

Contracts I signed with a company of Marquel Russell’s, Global Wealth Firm, have non-disparagement language. My intent here is to communicate lessons learned after (1) my experiences with his company, (2) conferring with a number of clients of his, and (3) researching the “turn your passion into a profit” business coaching industry.

Have you had a bad experience with a coach but not expressed yourself publicly because of contractual language? Consumers’ honest reviews are protected by the Consumer Review Fairness Act, regardless of contractual language. Writers for law firm Morrison & Forrester state in this gratitude-inspiring article:

The FTC has cautioned companies that a consumer’s pure opinion is not likely to meet the “clearly false or misleading standard.”

I certainly cannot offer legal advice but I believe if you have engaged in a transaction with a U.S. business, you are probably protected.

Marquel Russell review – online coaching

Attending the “Matrixx” by PGI – a poor decision (it seems)

It thus far appears that attending the Matrixx, organized by the Proctor Gallagher Institute AKA PGI, was the worst decision of my adult life.

The following is intended to be fair. I understand that some will interpret my poor results in this context as merely an indicator of my personal weaknesses rather than finding fault with any opportunity or it’s marketing.

Hey, maybe you’re right.

***********************************************

I paid $18000 or $18500 to become a consultant for PGI and the experience was not what I dreamed it to be while in the more open, and gullible, state of mind I adopted at the Matrixx. It was there that I committed to becoming a consultant.

Running through what we have now given PGI: I think my wife and I paid a total of $27,500 to attend the Matrixx, and then there’s the aforementioned consultant fee ($18,00), and then there’s a $4000+ non-refundable (why?) deposit for something called the “1% club” that she lodged into her heart but did not attend. We could not afford the full cost of that seminar.

I have certainly learned a lot from this painful experience and have met some interesting people. PGI is certainly not afraid to ask for money. The company charges an annual fee after becoming a consultant, and then attending the training in Toronto is required (complimentary but one has to pay for travel and a place to stay), and then if in about 8 months you do not meet some requirements such as securing enough clients then there is a fee to extend the deadline, and then in the Facebook group I have seen invitations by the company to pay for additional training.

So the company continues to charge people who are out in the world advertising for the company. This would be fine if not for my impression that few consultants do well in this profession, despite the opportunity being sold as a “business in a box” with the implicit likelihood of success one would expect from an offering by “America’s greatest prosperity teacher”, Bob Proctor.

Watch “Going To Zero” on Netflix before jumping into any “business in a box” opportunity. Many have lost a lot of money to Herbalife and it seems only a few have earned a nice compensation. Reminiscent of my current impression of PGI, people are dazzled into jumping into the opportunity by an appeal to their sense of awe. “Wow, I am a spiritual being who can do it all!” the person may think, and Bob Proctor says the right people and opportunities will come to you when you need them. I’m in!

Then you go home, are encouraged to take responsibility for your life, including the expensive business decision, and maybe you pull others into the opportunity, or maybe you blame yourself for not selling something you do not find sufficiently valuable.

This has been a humbling experience and I continue to work through my thoughts. I hope to publish more thoughts as I converse with other PGI consultants. I do know there are other disappointed consultants, maybe quite a few. Based on our weekly emails, it does not appear very many individuals are activating clients. Maybe 30 consultants per week activate a client out of the ~550 consultants, and I’m thinking many consultants reappear in almost each email. Setting up an analysis to confirm those numbers would take some time.

I know at least a few consultants love what they do, such as those repeater activators. My wife is still selling the personal development program we can sell as PGI consultants, which PGI describes (in my view) as a panacea for all : “Thinking Into Results”. Activating a client means getting someone to purchase the program (~$3000-$10000), which is expected to include some kind of consultation with the consultant – often a group Internet video-call but some consultants do one-to-ones.

The state-of-mind my wife and I adopted during and shortly after The Matrixx caused me to trust, again as Bob Proctor said, that the right people would come to me when I needed them. So we, perhaps in a poor interpretation of Bob’s ideas, jumped into two expensive business coaching opportunities to assist with Facebook marketing (more on that one in the future).

It turns out all three of these programs had redundant elements, particularly with respect to Facebook marketing and a sales technique that focuses on making a client believe you are an authority who has the answer for whatever key problem they have.

I am working towards forgiving myself. Part of that process includes helping others avoid the pain we have experienced while allowing that some may see opportunity in these words. My wife thinks “Thinking Into Results” (TIR) is ethical to sell to the right person. Conjoined with her consultation, TIR has helped at least two clients overcome important hurdles.

My wife and I certainly have some salient personality differences. I am not ready to explore that topic yet.

Some people attend The Matrixx multiple times. I may correspond with a few of those individuals and comment on what I learn. My guess is that Matrixx attendees that are happy with the $15,000 cost ($12,500 as I recall for a second member in a pairing) are those able and willing to adopt a worldview in which they feel less guilt about taking money from people in exchange for a good, service, or opportunity that [EDIT: a personality like mine would predict] would likely lead to disappointment. The satisfied attendee, in my clumsy generalization, may not even think about probabilities. “Statistics is for losers,” the celebrity Steve Harvey once said in an inspiring speech that suggested he was a believer in the Law of Attraction.

The Law of Attraction: your thoughts attract the results you want in life by the sequence (1) thoughts, (2) feelings, (3) actions, (4) results. This is one of the supposedly universal laws presented in “Thinking Into Results”, the popular success tome “Think And Grow Rich”, and other similar works.

Sounds good, right? Visualize yourself with a large amount of money and you will take the actions to earn it. Fail to earn the money you wanted? Your fault. Your results reflect your thinking by this law. And it’s a universal law, so you can look across history and space and apply it. Holocaust victims and trafficked children, did they just have the wrong thoughts?

I hope I have this wrong.

Watch for a philosophy that communicates paramount individual responsibility and a devaluing of concern for others. This philosophy is strong in “personal development” circles including those of network marketing, “law of attraction” devotees, and “business coaching” as a category. Practitioners will say things like, “Focus on the use value of what you are offering”, which means focusing on the potential longoterm impact of a good, service, or opportunity even if in the short-to-middle term, most experience disappointment. The freeing feature of such a philosophy is that one can always say: Eventually, a person or company that pays you for something will derive more value than they paid. You do not have to see this play out, just trust. You also need not ask how people are doing. Just keep adding clients and trust the process.

“Spirit is always for expansion and fuller expression.”
– Bob Proctor

By that philosophy, perhaps I should thank PGI and those two business coaches for taking so much of my money and making me a more cautious consumer. I will say that PGI and the two coaching companies did offer something of motivational value, and if all had worked as my wife and I had hoped, maybe I would not be typing this post. However, given what I know now, I am sure that I would still be questioning whether my most ethical action would be writing my impression that most will not get a sufficient return on their investment into:
1. the Matrixx
2 becoming a PGI consultant
3. purchasing business coaching from anyone who dubiously calls him or herself “the most sought after digital marketing coach on the planet”. I know, I know…I made an embarrassing decision.

What I do not recall any of these three entities (PGI + two coaches) expressing is that I was investing in a high-risk opportunity given, as far as I can tell, most investors fail miserably. That “most sought after…coach” actually said in his advertising that most of his clients earned back more than their investment in the first month, which would have been $6000. I am extremely doubtful about that claim’s truth but I cannot conclusively falsify it. Perhaps authorities will.

Concealing and / or ignoring historic failure appears to be a persistent element in business. For example, most restaurants fail and yet people develop the faith to keep trying. People approach others about business opportunities all the time without communicating risk of failure unless pressed, and even then they may bias their remarks towards the optimistic.

We need to speak more often about ethics to avoid distrust fracturing our system.

Great confidence should not be assumed in these words because I am asking myself and others a lot of questions. I am appreciating more than ever the power of faith and indeed, if one maintains a belief in ideas such as:

“I am a creator like God so I can create the reality I want for myself” (paraphrasing Bob Proctor)

then a powerful motivation to take risks will be sustained.

I overhear my wife helping others to maintain their belief in their dreams and keep taking action. It sounds helpful and righteous.

On the other hand, this belief can also lead others to blame themselves and act destructively when things do not go well:

“Something must be wrong with me. I thought this would fix my problems.”

Exacerbating the problem, the communities that form around what I’ll call “coaching programs” have descriptions that speak to being positive and there’s the threat of being kicked out for being otherwise. This helps explain why I have rarely seen complaints or heavy-hearted concerns in those communities (Facebook closed groups in particular). In talking with others and sharing my shameful experiences, I have discovered others’ pain. Our shame has lightened through conversation.

Speaking to the healthiness of acknowledging negative emotions, a fellow PGI consultant today mentioned how helpful the book “The Subtle Art of Not Giving a F…” by Mark Manson was in healing her pain after having invested so much in the PGI consultant business venture and having told friends and family of her upcoming material success. She described Manson’s work as the “anti-self-help book for the ages”, a book that encourages one to acknowledge the full spectrum of their emotions and then with that more balanced sense of self, focusing one’s energy on their priorities. I loved her adoption of values less focused on material wealth and more on what work she would find holistically valuable at this time, teaching in schools.

There’s an opportunity here: A personal development program that prominently connects (1) material success to the (2) spiritual while communicating (3) ethical values and (4) social interconnectedness. When comedian Dave Chappelle said in a recent interview, “We are either fighting with each other, or against each other”, it did not sound like a false dichotomy.

***********************************************

Returning to The Matrixx: I do recall that there were a surprising number of people in attendance who were involved with network marketing. Common to network marketing, at least insofar as I have experienced that field, companies like the Proctor Gallagher Institute charismatically present material that stimulate the imagination of a life with greater wealth and therefore more happiness.

I say: Guard your dreams and your wallets.

I may have to delete this post because I am waiting to hear from a PGI leader whether my contract allows a consultant to post such thoughts. If I change my mind after further study and conversation and results, then I will offer my mea culpa.

I think all three of the programs into which I entered have contracts that restrict negative language although I read of a recent law that allows negative opinions to be expressed regardless of certain contractual language. I am grateful that a free country allows consumers to know about others’ experiences.

I hope this post contributes to a better assessment of PGI, “Thinking Into Results”, business coaching, any kind of marketing coaching, and investing thousands to become a consultant for the Proctor Gallagher Institute or attend their Matrixx seminar.

Attending the “Matrixx” by PGI – a poor decision (it seems)