How Form search may fail

When you search on Find a Form, search is using an API (more info below) to return search results directly from the Forms DB. This search is not returning results from Drupal. If the Forms DB is down, or if the API is very slow, or unresponsive, users will be presented with an error, instead of search results, example below.

What is an API

API stands for Application Programming Interface. This just means: a standard protocol for how two machines or databases can talk to each other. In this example, the Drupal CMS talks to the Forms DB to receive search results and updated information, using an API.

How Downloadable PDFs may fail

The link to download a PDF points directly to the PDF file on a server. This could break in two ways:

1. The filename has been changed in the Forms DB

On Form detail pages, the Downloadable PDF link points to a filename. If the filename changes on the fileserver (and in the Forms DB), then the Find a Form search results will point to the old filename (that doesn't exist anymore) -- until the next Forms DB migration runs + the next content release runs.

Forms DB migration runs 3am ET nightly. Content-release runs at ~9am ET daily. This means that a filename change can cause a broken link from 3am until 9am (or first successful content-release of the day).

We provide steps below to troubleshoot this scenario. 

2. APIs are slow

Currently (Sept 2022), after a user clicks to download a PDF, our site calls to another API to validate the file. If the API we are calling is slow, this request may timeout or return a failure to validate the link. Users will then see an error. 

Bad PDF Link emails

The error above includes a link to "Email the forms managers." If a user clicks this link, their email client will open, &  create a draft email with Subject, "Bad PDF link." The email draft includes the form name, and a link to download the PDF directly from the Forms DB. <to do: clarify why it is we think this link will be correct. Or: it's not, in cases of filename change, only in cases of API latency?>

Forms Managers will receive this "Bad PDF link" email, including the Form name & URL. They typically report this directly to Dave Conlon. If you receive a "Bad PDF link" email, please copy/paste the contents of the email to the CMS helpdesk support form. The CMS helpdesk will help to triage the report for follow up.

Site users will see a PDF link in the email they send, which they can use to download the PDF file from the Forms DB. 

Helpdesk triage for Forms

Step 0: Reproduce

Try to reproduce the failure, whether it's search results not returning correct forms, or a PDF that won't download / returns an error. If you aren't able to reproduce, this may happen if a slow API recovers which fixes the problem. Ask customer to confirm they can still reproduce the problem / error.

Step 1: Verify the form doesn't download, & if the form is internal or public

Our team only supports public forms on va.gov. Forms on vba.va.gov are owned by the Forms DB team. Check the Form URL provided:

• If vba.va.gov (internal) - ask the Helpdesk requester to contact Kevin Reid, Kevin.Reid@va.gov, and share the details with him.
• If va.gov (public) - Proceed to step 2

Step 2: Confirm or Rule out filename or metadata changes

1. Log into Prod CMS
2. Check the Forms Flagged Content Dashboards in the CMS: 
   1. Changed title 
   2. Changed filename 
   3. New / Deleted
3. Search for the reported form name. 
4. If the reported form(s) do not appear, jump to Step 3.
5. If so, click the form name to see node view
6. In the Form node, in right sidebar, under Recent Changes, see a description of any change. This may be enough to know what to do next.
7. If you need more detail / confirmation, click the Revisions tab & select the two revisions you want to compare
8. Scroll down and click Compare selected revisions
9. Under Layout choose Split fields
10. See the details of the change. 

Step 3: Follow up on changed data

If a filename has changed:

Form managers have been instructed not to make name changes unless absolutely necessary because it causes the form to lose ranking on search engines, and can also break links that point to the form PDF from other parts of VA.gov.

To resolve:

1. The link will stay broken until content-release runs, so visit https://github.com/department-of-veterans-affairs/content-build/actions/workflows/content-release.yml to verify if Content Release has deployed successfully for the day.
   1. If not, run the workflow or talk to CMS Platform team to trigger a release. This will update the user-facing link to point to the new filename, preventing new reports of a broken link. 
2. Email the Forms Manager who reported the Bad PDF Link emails, and/or the Forms manager who owns the form (whether NCA, VBA, VHA, VEO). <TODO: identify who is who> Notify them that the filename changed, and request they revert to the old filename. Ask them to notify you as soon as filename is reverted.
3. When filename has been reverted, that change will sync to Drupal DB at 2am via the next Forms DB migration. A content-release should run immediately after, to update the user-facing link again, to point to the restored original filename. If it doesn't run, repeat step 1 above.

If other metadata has changed:

We have seen at least one instance of swapped metadata in the Forms DB causing issues (#9682).

If the filename has not changed, but other metadata has changed for the reported/problematic Form, file an issue for Public Websites team to review: (PW) Forms/Bad PDF link defect . Include details about the Form, how you received the report, and your research so far.

If you can reproduce the error, but research gives no clues:

File an issue for Public Websites team to review: (PW) Forms/Bad PDF link defect