| Are the objectives, scope, and methods for this review clearly described? |
|---|
| 1 | 2 | Yes | None |
| 2 | 3 | Yes | None |
| 3 | 4 | Yes | None |
| 4 | 5 | Yes | None |
| 5 | 6 | Yes | None |
| 6 | 7 | Yes | None |
| Is there any indication of bias in our synthesis of the evidence? |
|---|
| 7 | 2 | No | None |
| 8 | 3 | No | None |
| 9 | 4 | No | None |
| 10 | 5 | No | None |
| 11 | 6 | No | None |
| 12 | 7 | Yes - Heavily focused on Virta and comparing it to other interventions, and making recommendations for VA studies with Virta. The scope and questions don’t match much of the Recommendations and Conclusions. | We broadened the discussion and conclusion sections to discuss all the virtual diabetes programs identified in the report. |
| Are there any published or unpublished studies that we may have overlooked? |
|---|
| 13 | 2 | No | None |
| 14 | 3 | No | None |
| 15 | 4 | No | None |
| 16 | 5 | No | None |
| 17 | 6 | No | None |
| 18 | 7 | No | None |
| Additional suggestions or comments can be provided below. If applicable, please indicate the page and line numbers from the draft report. |
|---|
| 19 | 2 | This is exceptionally useful and timely information for VHA’s ongoing evaluation of the Virta program. The results are clearly and accessibly presented and the recommendations are well grounded in the evidence. | Thank you. |
| 20 | 3 | This well-written and engaging report describes the results of a nicely-conducted rapid review of virtual diet programs for patients with type 2 diabetes (DM2), which has a special focus on the Virta Health program. Overall, the findings appear valid and the interpretations reasonable. I have the following comments, organized by section. | Thank you. |
| 21 | 3 | Intro 1) Page 8, line 36 – Change “replace insulin or modify how insulin is used by the body” to “augment insulin levels, increase sensitivity to insulin, or impart other glucose-lowering effects.” | We made this change. |
| 22 | 3 | Intro 2) Page 9, line 15 – consider changing “limits body’s production of glucose” to “limits body’s access to glucose” to account for decreased enteric absorption of glucose as well as ‘production’ (implies gluconeogenesis). | We made this change. |
| 23 | 3 | Intro 3) Page 9, line 21 – Comment on calorie restriction. Some ketogenic diets do not require aggressive calorie restriction, which can also make them easier to follow for some. | We added: “and there are no restrictions on the amount of calories consumed.” |
| 24 | 3 | Intro 4) Page 9, line 24 – Consider noting potential for acutely worsening LDL with ketogenic diet under ‘skeptics’ points. This may be idiosyncratic, in that certain patients can experience dramatic worsening of LDL. See: KirkpatrickCF, et al. Review of current evidence and clinical recommendations on the effects of low-carbohydrate and very-low-carbohydrate (including ketogenic) diets for the management of body weight and other cardiometabolic risk factors: A scientific statement from the National Lipid Association Nutrition and Lifestyle Task Force. J Clin Lipidol. 2019Sep-Oct;13(5):689–711.e1. doi: 10.1016/j.jacl.2019.08.003. Epub 2019 Sep 13. [PubMed: 31611148] [CrossRef] | We added “it is unclear whether these benefits are maintained over time, and if there are any long-term risks to patients with diabetes such as worsening of cardiovascular disease risk factors including LDL cholesterol” and added Kirkpatrick 2019 as an additional citation. |
| 25 | 3 | Intro 5) Page 9, line 26 – With appropriate medication management, hypoglycemia risk may actually be lower with ketogenic diet – see: YancyWS, et al. Comparison of Group Medical Visits Combined With Intensive Weight Management vs Group Medical Visits Alone for Glycemia in Patients With Type 2 Diabetes: A Noninferiority Randomized Clinical Trial. JAMA Intern Med. 2019Nov4;180(1):70–9. doi: 10.1001/jamainternmed.2019.4802 [PMC free article: PMC6830502] [PubMed: 31682682] [CrossRef]. Online ahead of print. Consider mentioning with potential benefits. | Thank you for your comment. We decided not to include reduced risk of hypoglycemic events as a potential benefit of the ketogenic diet. Although the Yancy 2019 study you cite did indeed find a reduced risk of hypoglycemia in the intervention arm among those that consumed a low carbohydrate diet, that arm also attended more frequent group visits and received other co-interventions such as physical activity and weight management counseling, so we cannot conclude that it was the diet that caused the reduction in hypoglycemic events. The Kirkpatrick 2019 review you cite above also notes that the ketogenic diet may be associated with an increased risk of hypoglycemic events (and thus patients should be monitored closely and have medications adjusted as needed, as you point out). |
| 26 | 3 | Intro 6) Page 10, line 11 – Additional info on the approach to “physician management of medications” would be helpful. | We added “(ie, titration of medications based on biomarker tracking)” to give a brief overview of the physicians’ approach to medication management in this study. |
| 27 | 3 | Intro 7) Page 10, line 16–21 – Include info on how patients were identified for Virta pilot. Was this a voluntary, opt-in design, and what would that suggest about selection bias and generalizability of findings from this open-label, uncontrolled study? | For brevity’s sake, in the executive summary we comment that the study has “important limitations” and then provide more detail on these limitations (including the fact that participants chose which intervention they wanted to participate in) on p. 20–21 of the report. |
| 28 | 3 | Methods 1) Inclusion criteria – Only included last 5 years, but given evolution in technology, reasonable to assume that relevant studies would be captured in this window. Limitation further mitigated by scanning reference lists and consulting with experts as a quality check to assure no missed articles within or prior to window. | No comment. |
| 29 | 3 | Methods 2) Reviews at title/abstract and full text level were by one reviewer with overreading from another for this rapid review. Customary for 2 investigators to independently review each citation at these levels, with citations moving to the next level when included by either reviewer. Though reasonable for a rapid review approach used for this project may have reduced sensitivity. | No comment. |
| 30 | 3 | Methods 3) Data abstraction completed by one reviewer with overreading from another – no concerns. | No comment. |
| 31 | 3 | Methods 4) Quality assessment performed using a validated tool by one reviewer with overreading from another – often QA is done by two independent reviewers, but utilized approach likely adequate for rapid review. | No comment. |
| 32 | 3 | Methods 5) SOE assessment appropriate | No comment. |
| 33 | 3 | Methods 6) Given conceptual heterogeneity in included studies, qualitative synthesis appropriate. | No comment. |
| 34 | 3 | Methods 7) Included outcomes appropriate, as were definitions of clinically meaningful changes. Might have also considered ascertainment for other patient-centered outcomes such as QOL, Diabetes Distress, etc. | We agree that these are important outcomes and have included quality of life as an additional outcome that studies should evaluate in the future (p. 35). We also added language to the “limitations” section (p. 34) to indicate that examining these 4 outcomes alone was a limit of our review and future reviews should examine “other patient-important outcomes associated with these programs, such as patient satisfaction and quality of life.” |
| 35 | 3 | Results 1) Page 19, line 51 – Allowing patients to self-sort into Virta group vs. control is a MAJOR weakness of the approach in this study. Not only does this design feature likely underlie the measured differences between the intervention and control groups (baseline differences in BMI and insulin use), but it is likely to have introduced innumerable between-group differences in unmeasured factors like motivation, comfort with technology, medical complexity, and others. In essence, this study appears to have allocated people who wanted to lose weight to Virta and those that didn’t to control. In light of this selection bias, I would hesitate to treat this study as truly controlled, and instead would consider it as a pre-post examination of Virta in a highly selected population (meaning that the generalizability of the findings to the wider Veteran population is likely very limited). I see that these and other issues are noted in the ‘Limitations’ section on page 20, line 52; the criticisms in the first paragraph of this section are appropriate, and if anything, could be even stronger. The concerns articulated re: changes to the clinicaltrials.gov protocol and data fragmentation are also highly concerning. | We agree that the non-randomization of patients into intervention and control groups and data fragmentation are major limitations of this study. We did not make any changes to the report based on this comment. |
| 36 | 3 | Results 2) Page 20, line 41 – Would clarify “Overall, these results indicate the benefits of the program are maintained long-term” by adding “Overall, these results indicate the benefits of the program are maintained long-term in this selected population.” | We made this change. |
| 37 | 3 | Results 3) Limitations for other studies appropriately noted. If possible, would be helpful to have additional data on how populations were recruited for the other studies (TeLiPro, Low Carb Program, Better Therapeutics, etc.), as this information would help in considering the external validity of these approaches. | We added information on participant recruitment processes for each study. |
| 38 | 3 | Summary/Discussion 1) Page 32, line 18 – This is a rather generous interpretation of the Virta study, given the concerns about selection bias and lack of causality raised by the investigators in the Limitations section associated with that study (page 20, line 52). Might consider rephrasing as: “The study of Virta Health had critical limitations, but does suggest that for selected patients, participation in Virta Health is associated with improvements in important diabetes outcomes (weight, HbA1c, medication cessation, and diabetes reversal).” The authors indicated that benefits “were associated” with participation in the other named diets (page 32, line 25), so would certainly use the same cautious language for the Virta study. | We agree that it is important to use the same cautious language in describing the Virta study given its limitations, and have revised this sentence to say: “Though the study of Virta Health had important limitations, it suggests that for selected patients (ie, those who are severely obese, interested in an intensive diabetes management program, and willing to adhere to the ketogenic diet), the Virta Health program is associated with improvements in diabetes outcomes such as weight and HbA1c. Some patients who participate in Virta Health also stop taking medications and reverse their diabetes (ie, reduce HbA1c <6.5% with no medications or just metformin).” We also revised a similar sentence in the executive summary. |
| 39 | 3 | Summary/Discussion 2) Given that most of the relevant studies of technology-facilitated named interventions were not RCTs, there is a similar concern re: page 32, line 51. Rather than “2) diet counseling from a health coach can lead to clinically meaningful improvements in diabetes-related outcomes- but the use of technology to facilitate tracking of health data or increase the number of touchpoints with a health coach can lead to additional improvements,” would say “2) diet counseling from a health coach may be associated with clinically meaningful improvements in diabetes-related outcomes- but the use of technology to facilitate tracking of health data or increase the number of touchpoints with a health coach may be associated with additional improvements.” | We agree and changed the language from “can lead” to “may be associated with.” |
| 40 | 3 | Summary/Discussion 3) Agree with discussion of limitations of comparing multi-component interventions to UC (page 33, line 32). This suggests that a truly convincing study of Virta (or other named diet) would either use a similar virtual platform to compare two different diets (e.g., LCD vs. Mediterranean, given VA/DOD initial recommendation for Mediterranean diet) or use different approaches (e.g., virtual vs. in-person) to compare the same diet. A factorial design could examine both the diet type and delivery model, but would require a larger, more expensive study. | Agreed that a factorial design would be an ideal study design to address both questions on diet type and delivery model. However, given the short time frame in which the prospective evaluation would need to start, we have focused our research recommendations on what we believe is the most important question to address (ie, whether ketogenic diet is necessary, or if another diet can be used instead). |
| 41 | 3 | Summary/Discussion 4) Page 34, line 1 – In addition to suggested covariates, would also suggest that validated scales be used to capture baseline and longitudinal information on key factors like motivation, diabetes distress, quality of life, as these may represent important moderators of intervention effect. | We added a sentence: “Researchers may also consider measuring other baseline characteristics such as motivation or comfortability with technology, as this may provide additional information on mediators or moderators of treatment effect” to the “Recommendations for prospective evaluation” section. |
| 42 | | Summary/Discussion 1) Page 35, line 8 – Unless VA has expressed that it is committed to Virta to the exclusion of other options, would hesitate to make the following recommendation: “A second approach would be to use a non-Virta program as a “back-up” in the event participants could not tolerate the ketogenic diet or otherwise did not like the Virta program.” Even with appropriate measurement of and adjustment for baseline factors, comparing front-line Virta users to users of another program who did not like or tolerate Virta would fundamentally be an apples-to-oranges comparison, and would not answer the question of which program works better for Veterans. Would only use such a design as a last resort. | Agreed and we added the sentence: “This option should only be used if it is not feasible to let participants select their preferred program.” to clarify this should be a last resort. |
| 43 | 3 | Conclusions 1) Page 35, line 24 – Concerned about the validity of this statement as per above comments on Page 32, line 18. Would instead couch this statement in terms of “select populations” and “association” (rather than language that even cautiously implies causation). Existing data on Virta does not clearly establish causation, and the magnitude of the findings should not be generalized to the wider, unselected Veteran population. | We revised the conclusion to discuss all virtual diabetes programs in response to another reviewer’s comments. However, we removed any causal language and instead used language that describes that selected participates may lower diabetes outcomes after participating in intensive diabetes management program based on the ketogenic diet or other diets. |
| 44 | 3 | Conclusions 2) The fact that the findings from the existing Virta study should not be generalized to the wider, unselected Veteran population (particularly in terms of that magnitude of effect) means that these data have major limitations in determining the cost-effectiveness of the Virta approach for VA overall. May consider mentioning this. | Agreed and we added the sentence: “Additionally, because studies enrolled participants who were interested in intensive diabetes management programs and met other study eligibility criteria, findings may not apply to the wider, unselected Veteran population.” |
| 45 | 3 | Conclusions 3) In order to assure applicability of future findings re: Virta to wider Veteran population, studies are needed in representative populations attained via proactive sampling (not simply those choosing Virta, as this will bias sample toward individuals likely to benefit), active comparators (including non-LCD options) or other appropriate control groups, randomized designs, longer timeframes, and a wide range of clinical and patient-centered outcomes and harms. Could consider broader statements to this effect in Conclusions (recognizing the real-world fact that RCTs may not be feasible prior to adoption – however, there are major concerns about the current level of evidence supporting Virta). | We disagree that to assure applicability to Veteran populations, patients must be randomized to interventions. We believe the most appropriate strategy to determining the real-life effectiveness of Veterans choosing which diet or program they want to participate in is to use a non-randomized study design. In the “future research needs” section, we comment on the need for non-randomized study designs that evaluate active comparators and a wide range of clinical and patient-important outcomes and harms. We therefore have not made any revisions to the conclusions to address this comment, as the most important points are covered in the “future research needs” section. |
| 46 | 4 | This is a high-quality well-written review. I agree strongly with the recommendations made for evaluating Virta in both the retrospective and prospective group of Veterans. Having comparators will be quite useful and make the study findings much more useful. | Thank you. |
| 47 | 4 | It was not fully clear to me from a quick read how much of the data was from what was sent by the companies themselves since there was a note that data were requested from companies. I may have missed this in a quick read of the report. There are clear potential biases in companies reporting their own data and in studies published by companies. Please make sure to mention potential reporting bias in the executive summary and summary of limitations. This was already mentioned under specific studies but it was not clear how much this might impact study findings overall in the summary of findings. Apologize if I just missed seeing this. | We agree that there is the potential for reporting bias in describing data from literature that has not been peer reviewed. We have therefore revised the sentence on p. 14 to now state: “We have incorporated a summary of findings from relevant conference abstracts provided by Virta in the “Virta Health” section, but did not formally include these articles in our report.” We also previously stated in the Virta Health section on p. 20: “Additional data are available on participants in Virta Health’s non-randomized controlled trial via conference abstracts, but these data have not been peer-reviewed. We briefly discuss these findings here, but do not formally include the abstracts or evaluate study quality given the more limited information available in abstracts. Readers should interpret these results with caution.” |
| 48 | 5 | Appreciate this review offering future suggestions and ideas for research and data gathering for Virta and VA data collection. Suggestion I would offer includes: if possible, please be certain to make clear the direction ADA and VA/DoD suggest in regards to diet. They both support individualization of diet/nutrition per each person’s preferences and needs. Along with individualization for diabetes self-management. The Mediterranean diet was referenced as another option for patients to try, but bottom line it needs to be made clear that their recommendations include ongoing individualization. | We added the sentence to the description of the VA/DOD recommendations: “These recommendations emphasize that the chosen diet be tailored to patient preferences and needs.” |
| 49 | 5 | The reviewers may also consider risks such as CAD and CKD in addition to diverticulitis as mentioned towards end of this review. CAD and CKD as a whole may have a much deeper impact and create risks for these diabetic patients in the long term if following something like the ketogenic diet. Perhaps a study showing this evidence may need to be included or considered. | We added coronary artery disease and chronic kidney disease as additional conditions that should be monitored in future studies. |
| 50 | 5 | Greatly appreciate being a part of this project! An excellent review by the ESP team! Thank you. | No comment. |
| 51 | 6 | First, I really appreciated being an informant and reviewer for this report. Thanks for the opportunity! Second, this report is excellent. It is very informative and well written. I’m so happy you guys looked into this data. I especially like the recommendation regarding further research to figure out if it’s the diet that’s helping or the program itself. As an educator, I’m really excited to get an answer to this, so thank you! | No comment. |
| 52 | 6 | Only 1 edit found: Dietitian is spelled wrong in 2 places (page 24, line 23 and page 28 line 39) | Thank you, we have corrected these. |
| 53 | 6 | Other comments (all subjective and from an educator’s standpoint, so feel free to take them or leave them): 1. Page 7, states “we recommend that researchers capture a wide range of information on harms, including exacerbations or development of conditions such as diverticulitis”. Completely agree with this statement but there are more severe and common diseases we worry about and see with the Keto diet. So I think a better example would be something like kidney failure. Your line isn’t wrong, just a suggestion. | Per comment #49, we added chronic kidney disease as an additional condition that should be monitored in future studies. |
| 54 | 6 | Other comments (all subjective and from an educator’s standpoint, so feel free to take them or leave them): 2. Page 32 under Summary and Discussion: words such as “convincingly” and “rapid”, how are they defined? Without a definition it almost sounds biased. | Per comment #38, we have changed this sentence to say: “Though the study of Virta Health had important limitations, it suggests that for selected patients (ie, those who are severely obese, interested in an intensive diabetes management program, and willing to adhere to the ketogenic diet), the Virta Health program is associated with improvements in diabetes outcomes such as weight and HbA1c. Some patients who participate in Virta Health also stop taking medications and reverse their diabetes (ie, reduce HbA1c <6.5% with no medications or just metformin).” |
| 55 | 6 | Other comments (all subjective and from an educator’s standpoint, so feel free to take them or leave them): 3. Page 32: The paper says several times “patients who are unlikely to improve in usual care”. How is this defined? From an educators standpoint those that sign up for Virta seem to me to be the kind of patient that would benefit from usual care, so I’m curious. | We agree this phrase is confusing. Throughout the report, we removed “patients who are unlikely to improve in usual care” and replaced it with “for selected patients (ie, those who are severely obese, interested in an intensive diabetes management program, and willing to adhere to the ketogenic diet) to make it clear who might benefit from this program based on the existing evidence. We have also added a sentence to the “future research needs section” to indicate that: A third alternative would be to compare a commercial program to continuous care provided by a diabetes educator within the context of an interdisciplinary care team, an intervention that is similar in intensity to Virta and other commercial programs but is not delivered virtually. The VA’s National Center for Health Promotion and Disease Prevention has expanded a program of Telephone Life Coaches who may be able to deliver a comparably intensive program.” |
| 56 | 6 | Other comments (all subjective and from an educator’s standpoint, so feel free to take them or leave them): 4. The report says the Virta program results in decreased weight, HbAlc, medication cessation, and diabetes reversal. You define “diabetes reversal” as and HbAlc <6.5. I would add HbAlc <6.5 without medications because diabetics on meds/insulin can achieve an HbAlc <6.5 without reversing diabetes. Also, not all T2 diabetics can reverse diabetes or get off of their meds. If their diabetes has progressed to the point that they no longer produce enough insulin, medication cessation and diabetes reversal is not possible. Only those that still produce their own insulin can achieve this. I worry it gives a false hope. I would think a statement more like- Virta leads to weight loss and decreased HbAlc and can potentially lead to medication cessation and diabetes reversal in some, is more accurate. Again, that’s just an educators point of view. | We agree and added “without medications” to the “definitions of clinically meaningful change by outcome.” We also comment in the report on how each study defined diabetes reversal. In terms of the language on the Virta study specifically, we agree it’s important to clarify that only some patients experienced diabetes reversal. We have revised to say: “Some patients also stop using diabetes medications and reverse their diabetes.” |
| 57 | 6 | Other comments (all subjective and from an educator’s standpoint, so feel free to take them or leave them): 5. Last subjective comment: Bottom of page 34 it talks about the potential of other diets if a Veteran prefers. I agree some Veterans may prefer another diet, but the Keto diet is also contraindicated in some pts. I think that should be pointed out too. | We revised this sentence to say: “…the ketogenic diet may be unappealing or contraindicated in some Veterans with type 2 diabetes…” |
| 58 | 7 | Page 4 line 56: The ketogenic diet is not necessarily </=10% kcal from carbohydrate, and can be less than this if needed for the patient to achieve ketosis. | We revised to say “The ketogenic diet is a low carbohydrate, high fat diet, where approximately 70% of an individual’s calories come from fat, 20% from protein, and 10% or less from carbohydrates.” |
| 59 | 7 | Page 5 line 14: The VA - Virta relationship in 2019 was a non-research Strategic Partnership, so you must remove the word “study” here and elsewhere when referring to the partnership. In this line and line 15, it is also unclear whether you are referring to the VA partnership or another non randomized study (add the reference # for the study to which you are referring). | We have changed “study” to “project” on line 14. We revised line 15 to indicate that a “separate” non-randomized study provides evidence on the Virta Health program. We do not include references in the executive summaries of ESP reports; however, we do include the references to the pilot project and study in the introduction section where these are discussed in more detail. |
| 60 | 7 | Page 8 line 43: The Academy of Nutrition and Dietetics does not abbreviate their name “AND” but rather The Academy. I couldn’t find another use of “AND” but it should be referred to as The Academy | We have changed “AND” to “The Academy.” |
| 61 | 7 | Page 9 line 6: The ketogenic diet is not necessarily </=10% kcal from carbohydrate, and can be less than this if needed for the patient to achieve ketosis. | We made the same revision here as in the executive summary: “The ketogenic diet is a low carbohydrate, high fat diet, where approximately 70% of an individual’s calories come from fat, 20% from protein, and 10% or less from carbohydrates.” |
| 62 | 7 | Page 10 line 21: The VA - Virta partnership is not a research study; please remove any reference of this partnership as a “study.” https://www.blogs.va.gov/VAntage/58037/innovative-treatment-vets-type-2-diabetes/ | We have changed “study” to “project” where applicable in this paragraph. |
| 63 | 7 | Page 10 line 30: This sentence seems at odds with the stated scope of the ESP. While perhaps true and very valuable, it doesn’t fit with the stated questions of the ESP. Recommend remove this sentence: “This rapid evidence review was commissioned by the VA’s Health Services Research & Development (HSR&D) program to help inform evaluation of the VA and Virta Health pilot program.” | We believe this sentence is aligned with the stated scope and questions of the review. The scope of an ESP review is informed not only by the key questions and PICOs, but also the purpose and audience of the review. We have therefore left in the sentence in the report, but revised it for clarity. |
| 64 | 7 | Page 33: Gaps and Future Research section. Remove or rewrite to connect this section with your ESP questions and scope. | We revised the gaps and future research section to be more generally applicable to research of all virtual diabetes diet programs. |
| 65 | 7 | Page 35 Conclusions section. Revise to connect this section with your ESP questions and scope. Or delete the first sentence would suffice. | We revised the conclusion to discuss findings of all virtual diabetes programs. |
