How to Break Something Critical

How to break something critical right before you go to Orlando for a week & How to clean up the mess.

Well, I really stuck my foot in the mud on this one.  All of you Solo Admins are more likely susceptible to this but we “Mavericks” aren’t the only ones to engage in this type of brash behavior.  I am betting that a good number of you have found yourself just hours away from beginning a vacation and you were scrambling to wrap up a number of projects so you would have less to come back to the following week.  You were writing new formula fields directly in Production or making quick changes to profiles because someone whined about their visibility.  You get that Disney-Brain and all you want to do is wrap out some quick wins to end on a good note.  This week, I will tell you how I cut a few corners, broke a handful of rules, paid the price for my fool hardiness and dug myself out of the hole.

So to understand what I did, you will need a brief understanding of some of my data structure.  I have two custom objects, Project Task and Model.  Hanging below these objects is another custom object called EQLines.  An EQLine will be a child record of either the Project Task or Model but never both.  Due to a lack of foresight on my part when we had a trigger developed for us, we occasionally get orphaned EQLines floating around in the datasphere.  It’s really not a big deal.  I just need to pull the IDs of those orphans and euthanize them on a quarterly basis.  Simple Spring cleaning, right?  Well, if I were just cleaning up these records, I would have likely done it right.  I would have pulled the report that I had already saved for just this occasion and removed them from the system with ease.

So Anyways, I was just revising a Process Builder that established a semi-unrelated lookup relationship DIRECTLY IN PRODUCTION… Did you all just here that?  That was the sound of thousands of Admins slapping their palms to foreheads.  I know, I know: but you must remember that I had a reservation to eat dinner with Belle and the Beast in their Castle.  Who has time to think about validating change sets?  Anyhow, to test this change I would need a report to confirm the process was running accurately.  BINGO, it worked perfectly! Oh, and now this report is giving me list of all those Orphan Records.  I’ll just use this list to remove them from the system.  One last thing to worry about, right?

This is where you yell at the movie screen.  “No Drew Barrymore!!  Don’t go outside!  Don’t rapidly delete a couple thousand records because this list ‘should’ be the same as the one you know is correct!”

DON'T GO OUTSIDE!!!
DON’T GO OUTSIDE!!!

All right.  Did you get it out of your system?  Good, because it happened and there is nothing I can do to fix this short of jumping in the DeLorean with Doc Brown.  Actually, I had this opportunity last week and I didn’t take it.  I was too focused on getting Doc to tell me whether he needed road where he was going.  Seriously, I never thought I would get so giddy about meeting a man pretending to be Time Traveler.

Where we're going, We don't need Roads.
Where we’re going, We don’t need Roads.

So if you didn’t catch on, my big mistake was when I deleted all those orphan records I also deleted ~1400 records linked to all the Models in our system.  My Engineer has graciously decided not to string me up for this mistake.  Now let’s review what can be done in recovering errantly deleted data.

The easiest place to find deleted data is the Recycle Bin.  Salesforce Help Doc: Using the Recycle Bin  It can be found at the bottom of the left hand menu.  The Recycle Bin has come in handy in the past when a User calls up crying that they totally didn’t mean to delete Record-X.  I would groan about how this will be a difficult rebuild and I may need some time to search the foundation data and I can’t guarantee every field has not been corrupted.  I imagine most of them know I making all that up; but if some of my colleagues believe I do literal data spelunking’ what’s the harm?  I could just show them where to find the Recycle Bin themselves, but I don’t want them to rely on a safety net when they delete data.  We are also small enough that I can generally handle these requests directly.  So why didn’t this work for me?  The recycle bin only holds so many records before it starts permanently removes the records.  It also has a time limit, so if this did not arise as an issue for two weeks, we could have lost all of it.  I found ~900 records but I really wanted to get all of them in one swipe.  I’ll set this to the side for now.

This next option is by far the best choice, IF you have done a bit of leg work up front.  How many of you have set up your Exporting Data Backup? Salesforce Help Doc: Exporting Backup Data  It is super simple and the best thing to protect yourself from any damage that those pesky humans can do to the records in your org.  Once it is set up, you get an email once a week or month that leads you to spot in Setup where you can download a zip file copy of your data.  What I learned this week, (or relearned.  Who knows, I set this up such a long time ago) is that you can set the export to only pick specific objects.  There is a simple check box that says “Include All Data”.  I have no idea why I didn’t check that box when I set this up.  Would you believe I never selected EQLines as something to be exported?  That means, in this case, my trusty data backup was completely worthless to me.  Please set up a weekly Data Export and for Kermit’s Sake, select “Include All Data”.  Don’t gamble on which records will be needed.

So where does that leave me?  I was about to go back to my Engineer with the good news that he only has to rewrite 1/3 of the EQLines in our system when it occurred to me that I have an X-App that I use to load my Sandboxes after a refresh.

For those of you who haven’t heard me blab on about X-Author, a brief synopsis:  X-Author is an app from Apttus that allows you to build excel templates which can pull and push data into a Salesforce Org.  It’s an amazing tool.  Look it up. AppExchange: X-Author for Excel

The EQLines are important for testing so I have copy of all the records loaded there.  The X-App was last updated the first week of this January.  YAYYYY!  I know that this was a one-off case.  Nearly no other person would have something like this to fall back on, but YAYYY anyway.  I massaged the data a bit.  The Recycle Bin Records will be the most up to date so I revived them first.  Then I compared that list to the list on my X-App.  This gave me another ~500 records to load.  The only records that are now missing are any EQLines written since January that were not found in the Recycle Bin.  I felt a lot better going to my Engineer with 98% of the data retrieved than any other option.

So what did we learn?  Set rules and boundaries for yourself.  I know it can be easy break the rules when you are the ref as well player, but they are there for a reason.  Use your sandbox and change sets.  The Recycle Bin is a great safety net but don’t rely on it.  Set up your Weekly Data Export.  Be lucky enough to have a copy of your records lying around.  Finally, be super nice to anyone who has to spend a day reviewing and rewriting records because you didn’t follow any of the points listed here.

So carry on Nerds, & learn from my mistakes or you will be destined to learn from yours.

3 thoughts on “How to Break Something Critical”

  1. I have discovered that it does actually pay to check your data export settings once in a while. I haven’t worked out when it happens, but the Include All Data checkbox UNCHECKS ITSELF!! Possibly when a new custom object is added, I am not sure. (I have just gone to look at mine and sure enough, it no longer has that box ticked….grrr….)

  2. Thanks for sharing your story! Another note about the weekly backup. If you add *new* custom objects after you’ve set up the weekly export, go back and add those, because the “all data” option only does it for the objects that are there when you set it up.

  3. Thanks Anna and Mary!
    This would explain why it appeared I didn’t select Export All in the first place. Very Valuable information.

Comments are closed.