As people who write about health and productivity, Coleman and I get tons of questions about supplements. You name it, we’ve gotten questions about it — nootropics, stimulants, mass builders, fat burners, the works. But no matter the compound, the questions that we get all boil down to the same thing: “Should I take it?”
The answer is simple but not particularly helpful: it depends.
Is it safe? Is there enough scientific evidence to say if it works, one way or another? If it does work, does it work well enough to be worth the cost? Does what it does matter to you?
There are a precious few supplements that really do help almost everyone, and you’ve probably heard of them already.1Fish oil and creatine are the big two in my book, plus Vitamin D if you live in a place where it gets below freezing in the winter. Everything else falls into one of a few camps: either there’s not enough data to know if it helps or not2Most essential oils and other traditional remedies., data showing that it doesn’t help unless you’re actively deficient3Most vitamins., or data showing it helps in a scientifically significant but practically insignificant way. There’s nothing wrong with taking anything in the latter two camps (provided that they are generally safe and don’t have side-effects), but most can’t be said to really do that much for the average person.
I love helping readers with these questions, but sometimes you’ve got to teach a man to fish instead of answering his questions about fish oil, so I decided to share the tools I use to evaluate supplements. What follows are the steps that Coleman and I both take to evaluate any supplement we’re considering, both for personal use and for our athletes.
Safety
First and foremost, we’ve got to figure out if the supplement is safe.
You would think that a supplement for sale online or in a retail store would be safe by default, but you would be wrong. Supplements, unlike drugs, are presumed safe unless proven otherwise, meaning that manufacturers can put little capsules filled anything they want on store shelves with no prior approval or oversight. Supplements are required to be pure and accurately labeled4This, of course, doesn’t mean much: pure and accurately labeled cyanide is still deadly if you eat it., but unless the chronically underfunded FDA can prove that a substance is harmful, it can be sold.
Thankfully, the National Institute of Health’s Office of Dietary Supplements puts out fact sheets on almost everything that you can buy in a store. If they say a supplement is safe, there’s no real harm in taking it beyond potentially flushing money down the drain, although be sure to make a note of the dose they call safe. More is only better to a point, and almost everything that’s beneficial in small amounts is toxic in large amounts, even necessary-for-life things like Vitamin A.
This is not to say that all potentially dangerous supplements shouldn’t be taken; you just have to understand the potential downsides and decide if the benefits outweigh the risks . Professional athletes decide to take high-risk supplements all the time because their careers rely on continued athletic performance — it’s worth it to them. For most of us who just want to feel and perform a little better on a daily basis, a bit more care might be warranted.
Lack of evidence
Our next step to take when evaluating an unfamiliar supplement (or when making a more detailed argument about one we are familiar with) is to pop over to examine.com and pubmed.com to do a little research and check out the science behind the supplement.
Pubmed is the firehose of science that you can read and summarize yourself. It’s extremely valuable to do this, especially for things without a lot of evidence, but it’s difficult and requires a certain amount of literacy in reading scientific studies.5We have an article with the basics of doing this in the works. This is why Examine is so helpful — their team of research reviewers summarize the data and explain it in a way that anyone who took high school biology can understand.
At this point, a lot of things get disqualified for lack of evidence. Essential oils are a good example; scientists have done very few high-quality experiments6We’re working on an article or two about what experiments count as high-quality, but the short answer is that high-quality experiments test to see if the supplement does the exact thing that people claimed it does in humans or (less ideally) other animals, in a controlled and blinded setting, a large number of times. testing if they actually do what they say they do (e.g.: make you calmer or more energetic or less prone to illness, depending on the oil).
An important note: getting disqualified here doesn’t mean that a supplement couldn’t work, just that we don’t have evidence that it does work — if it’s also safe, then again, there’s no real harm to taking it other than potentially throwing money down the drain.
Statistically significant
When there have been enough high-quality experiments done, we can then look at the data and decide if the results are significant.
High school science refresher: the definition of significance in science is narrower than the common definition. Scientific results are considered significant when there is little statistical chance that the results of the experiment could be explained by luck or random variation in the data. We measure that chance by measuring the the p-value7From the common statistical notation p = probability of the experiment, which is the probability that the experimental results could be explained by luck or random variation.8More accurately, p-value works by comparing two data sets, one experimental and one control. P-value is a measurement of the probability that the difference in results between the experimental group and the control group could be explained by chance. The lower the p-value, the less likely those results can be explained by randomness, and the more likely it is that the thing being tested is the thing that’s changing the results.
Experiments need to hit a p-value of p = 0.05 or lower (meaning that there is less than a 5% chance that difference between the experimental results and the control was just luck or random noise) to be considered significant enough to claim that the supplement (or whatever) actually works. A lower number is always better, and the more data being analyzed, the more likely the p-value can be trusted.9This is topic that’s outside of the scope of this article; in short, because p-values are a comparison of the means of two data sets, they only work well for things that follow a normal, thin–tailed distribution. Most biological processes do follow that kind of distribution, so p-value is fine for our purposes.
The result of this step is usually the same: for most common supplements, there is enough data from enough experiments to show pretty conclusively that they don’t do anything for healthy people who aren’t already deficient in that substance. Chromium is a good example: it’s very helpful for controlling blood sugar levels in people with diabetes and prediabetes, and a deficiency can create diabetes-like symptoms. Otherwise, it’s useless.10https://www.ncbi.nlm.nih.gov/pubmed/17519436
Statistically significant, practically insignificant
Sometimes, you’ll end up with a substance that actually does work, like the amino acid beta-alanine.
We produce beta-alanine in our bodies, along with another amino acid called l-histidine. Those two combine to form carnosine, a peptide that buffers acid in our muscles, keeping muscle pH stable for longer. Supplementing beta-alanine increases carnosine levels in the muscle, which helps the muscle stay at a functional pH for longer, allowing us to exercise longer.
In simpler terms: you know the burn you feel while doing high-intensity exercise? That burn comes from our inability to clear away metabolic waste in our muscles as fast as it’s produced, which happens when we’re working at high intensity. This build up of metabolic waste acidifies the muscle, lowering its pH and robbing it of power.11Fun fact: the metabolic waste that causes this acidification is hydrogen ions created in the breakdown of ATP, not lactic acid as usually believed Beta-alanine slows that process down.
There is a lot of high quality science that proves this conclusively, including this meta-analysis of all of the beta-alanine research done before 2012. Over and over again, experiments are able to show that people on beta-alanine are able to do short-burst (1–4 minute), high-intensity exercise for longer. According to all those experiments, the results are statistically significant.
We are 99.9% certain12All of this data comes from the the meta analysis linked above. We’re 99.9% certain the effect is real because in the analysis, p = 0.001. Saying p = 0.001 is the same as saying that there a 0.1% chance that the results are just random noise, and 100% – 0.1% = 99.9%. that supplemental beta-alanine improves work capacity. But that 99.9% certainty doesn’t tell us anything about how much it helps, just that it does help — by about 2.85%.
The improvement is statistically significant, but it isn’t practically significant. A 2.85% increase in work capacity means that your time to exhaustion is about seven seconds more over four minutes of exercise. If you were running in the final of the of the Men’s 1500M at the 2016 Rio Olympics, that’s enough to take you from 12th to the gold medal.13https://en.wikipedia.org/wiki/Athleticsatthe2016SummerOlympics%E2%80%93Men%27s1500_metres But for most of us, that improvement doesn’t mean anything to our day-to-day lives. Just because something is ‘scientifically proven to be effective doesn’t mean it has practical effect on our lives.
Putting it all together
- If there isn’t enough data to say anything about a supplement, including whether or not it’s safe: take at your own risk. Commonly used but poorly studied supplements (like essential oils and some traditional chinese and indian medicinal herbs) are probably fine simply because if they were harmful, we would have noticed by now.
- If you know it’s not safe, but also that it works: think very carefully if it’s worth it to you. The classic example here is steroids and other bodybuilding drugs. They work, but at a serious cost.
- If we can say it’s safe, but there isn’t data that says it helps people who aren’t already sick people: figure out if you’re deficient in it, and/or take it as insurance. Most vitamins fall into this category; this is why many doctors recommend taking a broad spectrum multivitamin as an “insurance policy”. Taking this kind of supplement is critical if you have an actual deficiency, though — Coleman and I both take methylated B vitamins because we can’t produce them naturally thanks to an MTHFR mutation.14We figured this out through genetic testing.
- If it’s safe, and there’s data that it helps healthy people: take it, but know that there are very few things that actually fall into this category. Vitamin D is one; it is critically important for a lot of biological functions, and most people are deficient. We produce it naturally via sun exposure, but anyone who works inside probably doesn’t get enough sun to produce an adequate amount of vitamin D, so supplementation helps.
- If it’s safe, and it works, but it’s not practically significant: decide if the improvement it offers is worth the time and money. Racetams and other nootropics/cognitive enhancers are the classic example — most improve memory and reaction time by a statistically significant amount, but they’re expensive, and the improvement is usually tiny. How much is a single percentage point improvement in memory worth to you?
An unearned generalization
I think this idea of true versus practically significant is relevant to most things in fitness and productivity, including mental models, actions, habits, supplements, or productivity apps. There are a few things that really do help everyone15Exercise, ubiquitous capture., a lot of things that don’t hurt but can’t be shown to help16In my opinion, most visualization/positive thinking/law of attraction stuff., even more that will hurt us if we’re deficient but not helpful to have a lot of17Vitamins, hygiene, todo lists, goal setting practices, a decent diet., and a multitude that can give us a 2% edge in something but are probably not worth the effort unless we’re trying to be world class in that thing.18 Most scientifically proven supplements, most specific productivity methods, most professional skill sets.
The trick is really figuring out what category any given activity//tool/whatever falls in, and then deciding if it’s worth it to you. Unfortunately, I’m still trying to figure out how to do that. More soon.
