Zombie metadata in custom (global) app

[James Bengel]

Just in time for Halloween, I have apparently found myself in the Night of the Living Metadata. And this is such a weird  edge case, that I'm not sure I'll find an answer here, but HI recommends putting the question to my perers first, because apparently we're quicker than they are.  So the weak of heart can turn back now.  Or you can take the red pill and see how deep the rabbit hole goes.

(TL;DR, I need to delete a custom app and there's metadata still hanging around making this impossible that I can't find.)

This all started with a decision both before my time and above my pay grade, which took a bunch of newbies down a twisted path to (eventually) building a custom app in global scope without the benefit of a development instance.  Our only available sub-prod instance was Test, and the results you can probably guess.

We had not at that time discovered the ability to create custom apps (in Studio!) in global scope, since all of our training to that point had been in a scoped app (which everybody knows as NeedIt).  So a LOT(!) of update sets got built in the process of trying to cobble this thing together.  Eventually the blind led the blind into the light, and the app was built, delivered, and put into production (and there was much rejoicing... yay...).  And the cycle of life as we knew it went form Test to Prod with all of us poking the bears at every opportunity to get a for-real Dev instance.

In the fullness of time, our relentlessness bore fruit, and our new Dev instance was built, and cloned ... from Prod.  Which nobody blinked at when we did it, because... well, truly, because we didn't know that's what they did.  But even if we had, the formerly mentioned app was stable, and nobody had needed to do much with it for a while, so nobody noticed that when you opened up Studio... there was nothing there. The app shows up in the repo, but it's not available to edit in Studio.

Some research eventually showed that this happens when you're working  backwards and cloning from an instance where the app was not built.  In other words (if I understood it right) had we cloned from Test, we wouldn't be facing this problem.  But we didn't, and we are, and now I'm trying to get rid of the installed app so that when we go back and clone from Test to Dev, there won't be a conflict that prevents it from putting down the app as it is in Test -- including (hopefully) the ability to edit the app in Studio. Now according to the documentation, you should be able to go to My Company Applications, drill into the app and click Delete, confirm it in a dialog and wait for the magic to happen. But you can probably guess it wasn't that easy, else I wouldn't be forced to admit all these embarrassing details. In fact, I was undone as soon as I tried to click the non-existent delete button. Because that is apparently not an option when you install backwards.  So how about uninstalling it?  Not any better, unfortunately:

Metadata still exists for this global application. Delete or move them to uninstall 'ERP Release'

(Note: We had a smaller, scoped, app that I was able to uninstall with no problem.  But I can't delete it either.)

I tried to get clever and create a new version of the ERP app (the big custom one) in Studio and then move all of the application files from the cloned version to the new version. That was pretty much a disaster, with a bunch of records ending up blank or corrupted and a couple of dozen more that were just missing for some reason.  But I was eventually able to rid myself of the "new" version, and then rolled back the old version and started again.  This time I figured I'd try the blunt force approach, and simply delete all the application files from the app and see what happened.  Seemed to be doing along okay, except there were (and are) 17 files that rise from the dead every time no matter what I do.  But here's the puzzling part: They're not actually there.  If you try to open one of them, you get the message "Record not found" and then you get to go back to the My Company Applications module, and wait for the spinners ot quit spinning so you can start over.  So I tried to back door the thing.  Went to sys_update_xml, found all of the records for the ERP Release app, and deleted them from there.  Had to sign off on the Sternly Worded Warning that this made things less upgrade safe, but that's kind of what I was trying to do, right?  No joy.  My 17 Zombie files are still there in the app.  Dug them up in sys_metadata and tried to delete them there.  Still have 17 files in the app that won't die.  And an app that won't uninstall.  And another one I can't delete.  Not even as security_admin.  I don't have access to be the admin admin (as in System Administrator "I Am Root" admin).  But if that would do it, I know who does have that access.  If I can tell him where to go and what to do to kill this thing once and for all.

[James Bengel · Accepted solution]

I had actually opened a case with HI for this at the same time I posted the question, since I had a feeling this was going to require intervention beyond my skill set.  And it appears I was correct. The gist of the case came down to this:

Investigation Summary: These 17 records in sys_metadata are orphaned records which is why you are unable to delete them. The related child table records is no longer present on the instance. On Test, these records do not seem to be orphaned. But in Dev, they were somehow not cleared from sys_metadata when the related child table records were deleted.

At the end of the day, the tech at Hi put in a change request to resolve this, but I have no idea what steps they took to do it.  But once it was done, I was able to uninstall the app cleanly, which I hope will be sufficient and there's not something else that has to happen in order to get the original problem solved.

I have a hunch that the "orphaning" was a result of trying to move all the existing application files form the old app to a new one that was created from scratch in Studio, but that's not even an informed guess.  As in I have nothing to support my hypothesis other than a lot of anomalies being created when I did that. And since there wasn't any explanation of what the Change request was requesting I'm not optimistic about ever learning the root cause.

But the solution seems to be to open a case with HI and let them deal with it.

[david631]

Your prose style is inspirational, Mr. Bengel. The pathos in each line lead me forward in breathless expectation of the next until, at last, I arrived at the culmination of the epic saga, and the Zombie-slaying heroes of the HI Service Portal stand wreathed in glory among the battered remnants of their enemies!

[Tim Grindlay]

Thanks for posting this, it was both informative and entertaining. 😂

I am in a somewhat similar situation and will let the folks at the hiPortal deal with it.

[MHager]

I have a similar issue and Support saying it can't be done of course.