Overview: AI Fixed 150,000 Lot Numbers In VAERS Data
Overview: AI Fixed 150,000 Lot Numbers In VAERS Data
Data Cleaning
Technical page link =>
Overview
Open source code and current results in data cleaning of vaccine lot numbers in VAERS (Vaccine Adverse Events Reporting System created by law in 1986 with currently about 1.25 million Covid-19 reports). Rather than a self-driving car, facial recognition or AI deciding which ad to show you, this uses some spell check routines, which is a form of AI. The most basic level of all artificial intelligence (AI) could be said to be the matching of outliers to known good groups and that’s what spell check AI does, like when you do a search for something online, the process is very good at figuring out what was intended usually.
Numbers here are as of April 22, 2022 data.
1,237,616 total rows/records of Covid shot reactions
‘Mere Text’ means no digit present so it cannot possibly contain a lot code
What the program does
Modifies outliers to closest sizable matching group with many each (since the large groups are likely all correct). Late-breaking development: Obtained a list of some known correct codes (~900+ currently) so those are all in the ok groups to match outliers to, and their expiration dates are an added column in the output csv file.
Screens out control codes entered mistakenly by users
Corrects manufacturers by pattern
(This includes about 1330 Astrazeneca codes (Europe, India etc). There is no provision in the system for entering reports involving that company, therefore people have been entering reports anyway and marking them as one of the other companies)
Creates a second distinct record for those times when people entered two lot numbers for a single report (quite logical of them since either could have caused the reaction or both)
Recovers over 19,000 lots from the comments field where the lot field is often blank or unrecognizable
Adds country identifiers to the state field (many reports are from outside USA)
Shows group counts before and after. For example, 039K20A went from 5291 of them to 6378 (+20.5%) by correcting typos like 39KZOA
Match score and confidence columns added
Removes control codes mistakenly entered like 214002, 59267100002, 000014A, URN:UVCI:01DE/I, J07BX03, PAA156051, CH.-B, BNT-162B2, NDC80777-273-10, EU/1/20/1528, 1110-20, MRNA-1273, 91301, about 40,000 of them altogether, and explains those in the code, sometimes with links for reference
Vax_Date column included. Any other columns of one’s choosing can also be made to appear in the output csv file.
It can add a Severity column based on some scoring assigned from the columns 'DIED', 'L_THREAT', 'ER_VISIT', 'HOSPITAL', 'DISABLE', 'RECOVD'. I gave death extra weight for example although we know those fields are not trustworthy, unfilled and filled incompleteness/inconsistency and no updates later.
Example, the typo 212C221A (instead of 212C21A) can be understood as part of the 53 reported dead of those 117 reports on that lot number.
Automatically combines the various VAERS csv files into one, filtered for covid, removing any duplicates/overlap year-to-year
Sidelines empty lots for a smaller working data file avoiding a row-limitation in spreadsheet programs for awhile. Maximum rows tend to be 1,048,576 (2^20) so these 1,237,616 rows would be too much without removal of empties.
Sort of an ungodly number of options like 20 (good or bad depending on one’s point of view)
Can simply extract strings of context surrounding some search terms in symptoms and quit
Can just save any desired columns together to csv and quit
Can save the Cleaned column from various runs in a file for comparison during development
Displays quite a bit of detail during a run (various verbose levels for even more detail)
Expiration dates for many (and thus known valid codes too)
Includes test mode functions built-in
Performs about 2.25 billion operations according the Python’s built-in profiler
How many of these lot numbers on vax cards can you make out?
You can see the problem above.
A round of applause for raw honesty by the one who entered “Huh?”.
Consider a scenario: Bob went to a clinic for the shot where one clinician read the lot number 039K20A off of the vial to another who had a moment of dyslexia writing it down as 309K20A. Bob goes home happy. The next week he’s in the hospital unconscious, the vax card is retrieved from his wallet by a doctor who reads the code to a nurse filling out the reaction report required by law. The folded crease and handwriting make the K look like 1C and so on. The doctor says 3O91C20A but the nurse hears 3O91C28 and types that into report lot field. 039K20A became 3O91C28.
People type EW5739 and say to themselves close enough, apparently. Some have fat fingers on the keyboard. Another 10 to 15% of humans are dyslexic evdinelty (they’ll be able to read that) where EW5730 and EW5703 look the same to them. Sometimes people add helpful things like 030H(OR4)21A and it’s a beautiful thing finding this code did the right thing.
These are 323 lot codes that all became 025J20A affecting 4105 records. Some can raise an eyebrow but are a better match than anything else.
| 025j20-2a | 025j20-2A | 025J20-2a | 025J20-2A | 025j20a | 025j20A | 025J20a | 025J20-2A / 025 | 025j202a | 025J202a | 025J202A | 025520a | 025520A | 025520-2a | 025520-2A | 025j20 | 025J20 | 025-j20-2a | 025-J20-2A | 02sj20-2a | 02sj20-2A | 02SJ20-2A | 25j20a | 25j20A | 25J20A | 0255202a | 0255202A | 02sj20a | 02SJ20A | 025j2oa | 025j20-24 | 025J2oA | 025J2OA | 025J20-24 | 25j20-2a | 25J20-2A | O25J20-2A | 025J02A | 025J20-za | 025J20-ZA | 025J20-20A | 025J20ZA | 025j2d-2a | 025j20/2a | 025j20-a | 025j20-219 | 025j20-214 | 025T20A | 025J2D-2A | 025J208 | 025J20/2A | 025J20-A | 025J20-219 | 025J20-214 | 025jz0a | 025j0a | 025JZ0A | 025J0A | 025020A | U25J20A | Moderna 025J20- | MODERNA 025J20- | 02sj202a | 02SJ202A | 025t20-2a | 025s20a | 025jZOA | 025T20-2A | 025S20A | 025JZOA | 025J204 | Mod 025j20a | MOD 025J20A | 025j20a-2a | 025j20-21 | 025U20a | 025U20A | 025J2o-2A | 025J2O-2A | 025J20A-2A | 025J20.2A | 025J20-21 | 025J20 2A | 025J0-2A | 025720-2A | 025-J20A | 025-520-2a | 025-520-2A | 025 J20-2A | O25J20A | O25J202A | 025s20-2a | 025j20-21a | 025S20-2A | 025J28 | 025J2024 | 025J202 | 025J20-2H | 025J20-21A | 025J20-20 | 025J 20-2A | 025020-2A | #025J20A | Lot# 025J20A | LOT 025J20-2A | J25J20A | D25J2DA | C25J2CA | 02SO20A | 025jz20a | 025j30-2a | 025j20=2a | 025j2019 | 025j2-02a | 025S202A | 025O20A | 025Jo2a | 025Jo2A | 025JZ20A | 025JLoA | 025JC0A | 025J30-2A | 025J29-2A | 025J20=2A | 025J2019 | 025J20-QA | 025J20-A2 | 025J20-@A | 025J20-2 | 025J2-02A | 025J 20A | 025720A | 02520A | 0251J20-2A | 025-J202A | 025-J20-ZA | 00J20A | moderna 025J20A | c25j20-2A | V25J2G2A | U25J02A | Q025j20A | OJ20A | O25J2O | O25J2D-214 | O25J20 | O 25J20-2A | Lot 025J20A | Lot # 025J20-2A | H02SJ20-2A | G2SJ20A | D2SJ20A | D2SJ20-2A | D25J20A | D255020A | C2SJ20-2A or 02 | C2SJ20-2A | 25J2OA | 25J202A | 25J20279.05 mc | 25J20/2A | 25J20 | 250J20-2A | 205J2OA | 205J20A | 125J20-2A | 0Z5J20-2A | 0UJ20A | 0NJ20A | 0N520A | 0MJ20A | 0M520A | 0J25J20A | 085j20-2a | 075T202A | 075J20A | 075J021A | 02sj20 | 02rj202a | 02YN20A | 02TJ20A | 02TJ202A | 02T520-2A | 02ST20-2A | 02SJZ0A | 02SJ20-A | 02FJ202A | 025v20-2a | 025q20-2A | 025jpa | 025joa | 025j2u-2a | 025j2021 | 025j202-a | 025j202(a or t) | 025j02-20 | 025X20A | 025W20A | 025V2049 | 025TA20A | 025T20A or 025G | 025T2032121 | 025T202A | 025STzo-2A | 025R 20A | 025LL0A | 025JZo2A | 025JZDA | 025JWA | 025JW-2A | 025JU20-2A | 025JO-2A | 025JLO-2A | 025JJ20A | 025JA20 | 025J70A | 025J520-2A | 025J2U4 | 025J2Q-2A | 025J2GA | 025J2CA | 025J29A | 025J26A | 025J25-2A | 025J20X | 025J20RA | 025J20J | 025J20D | 025J20A. | 025J20A-214 | 025J20A or T | 025J20A or 025T | 025J20A OR 02C1 | 025J20A EXP. 5/ | 025J20A CVS | 025J20A 028L20A | 025J20A 030L20 | 025J20A & 0 | 025J209 | 025J204A | 025J202SA | 025J202A/039K20 | 025J2025 | 025J20219 | 025J20214 | 025J202-2A | 025J201A | 025J2017 | 025J2014 | 025J201 | 025J200/025JZ00 | 025J20/2019(?) | 025J20-ZP | 025J20-VA | 025J20-UA | 025J20-JA | 025J20-7A | 025J20-2ALCHD | 025J20-2AA | 025J20-2A176912 | 025J20-2A0 | 025J20-2A. | 025J20-2A- | 025J20-2A and 0 | 025J20-2A Moder | 025J20-2A 12/31 | 025J20-2A 0 | 025J20-2A / pub | 025J20-2A (not | 025J20-2A 2120 | 025J20-25 | 025J20-219/4 | 025J20-217 | 025J20-1A | 025J20-02A | 025J20-02 | 025J20 A | 025J20 -2A | 025J2 0-2A | 025J028 | 025J020A | 025J00/NA | 025J-20A | 025J-20-2A | 025H20A | 025H20-2A | 025F20A | 025EJ2020A | 025D2049 | 025D20-2A | 025AJ20-2A | 025610A | 0255J20A | 0255J202A | 0255J20-2A | 02552022A | 025520/2A | 025520-A | 025520-2QA | 02550202A | 0253J20A | 02520J-2A | 02520J | 0252025J202A | 0251J20A | 025130A | 0250j20-2a | 025.J20.2A | 025-J20-219 | 025-520A | 025-20A | 025 j 20A | 025 J20A | 025 J202A | 025 J20-2 A | 025 J20 2A | 025 J020A | 025 J 20-2A | 025 520 2A | 0025J20A | 0025J20-2A | )25J20A | #025J20-2A |
(those like 025J20-2A which are valid codes are an issue I address on the technical page)
Before and After pattern counts per manufacturer showing the increase in counts
A bit of a video on it
https://www.bitchute.com/video/nsAebv5rORFD/
About Artificial Intelligence
Prior to the AI portion, the work is made easier for spell check with about 180 modifications by rule when they apply. If those are disabled then AI changes about the same number of records but it would require the most righteously superior among us to decide which of them are correct. This way the results are just more controlled.
Again at the root level, artificial intelligence is usually--arguably--just fitting an unknown to its best match among large pre-labeled-by-humans sets. The process is often accompanied by some lovely terms such as neural proximity contiguous ensemble correlation forest classifier covariance datum regression gaussian inertia cluster elliptical envelope matrices. Yeah well, this one’s spell check, nearest group.
In this case the large groups of lot numbers all entered exactly the same are considered valid, until there’s a published list of lot numbers from CDC and FDA. But also some sources of lot numbers with expiration dates have been obtained.
“An ounce of clean data is worth a pound of dirty data”
The fundamentals of AI, nice article: Stanford NLP Ph.D
"The simple approach beats the shiny one"
"accuracy grows approximately logarithmically with the dataset size"
"Never trust accuracies that seem like magic: anything above ~95 or 97%"
"Everything you read on the internet about ML **[Machine Learning]** (news, blogs, papers) is misleading or false until proven otherwise*
This code can never be perfect yet does have room for improvement and always will, no matter what.
There are people who cut logs into boards (that’s me) and others skilled at finish work, it would be helpful to have one of them apply their talent to this now.
It’s not uncommon to hear things like … tuning AI is more of an art than a science. I might also liken it to playing a musical instrument.
The idea is the guys that analyze VAERS data will take a look at these results. It is possible the greatest insight might be that their conclusions using the dirty data were already valid and that knowledge in itself would be useful.
Greater certainty should be the result either way.
The Reporting System
When adverse events are filed, the VAERS reporting form has a 15 minute timeout, or a writable PDF file can be used
"Healthcare providers are required by law to report to VAERS"
"Vaccine manufacturers are required to report to VAERS all adverse events that come to their attention."
Validating Lot Codes
Moderna expiration date lookup
Janssen expiration date lookup, the Johnson & Johnson subsidiary making the vax in Netherlands with partner German company BioNTech (pronounced like Byon-Teck)
Pfizer has no expiration lookup as of June 2022
The Bloomberg live vaccine tracker says 12 billion shots have been administered worldwide as of June 24, 2022. 594 million in USA.
Quick link to the number of deaths reported (29,031 as of June 24, 2022)
Doing great in life and could donate to my work?
I’m not asking for money from regular people but if you’re blessed with lots of extra money in life, you can make it possible for me do more of this type of thing.
Otherwise I’ll need to go to work for some company to make money.
For example if you bought a few thousand bitcoin for a few dollars each back in the day and could share some of those, it would be life-changing for the better here.
I will appreciate you forever.
Paypal: At the email below
Bitcoin: bc1qu7r6lztn97m5mqs8dcc936pnafnjyewsnwqtwh
To communicate …
