How to fix an odd LSBackup job error in Log Shipping?
My suspicion would be that even after removing the Log Shipping configuration the Log Shipping system tables are not being cleaned up properly. I would suggest the following steps from blog.sqlauthority.com for your situation:

1. Update the arguments for the command below and execute on LSPrimary01:

EXEC MASTER.dbo.sp_delete_log_shipping_primary_secondary
@primary_database = N'Primary_Database_Name'
,@secondary_server = N'Secondary_Server_Name'
,@secondary_database = N'Secondary_Database_Name'
GO


2. Update the arguments for the command below and execute on LSSecondary01:

EXEC MASTER.dbo.sp_delete_log_shipping_primary_database @database = N'Primary_Database_Name'
GO


3. Run the query from your question again and examine the results.

If there are still improper entries, consider manually removing them.  An example of something you could do would be this:

SELECT * INTO Tempdb.dbo.TemporaryCopy_log_shipping_primary_databases
FROM msdb.dbo.log_shipping_primary_databases 
--please note this only lasts until the database engine is restarted!

UPDATE dbo.log_shipping_primary_databases
SET monitor_server = 'LSPrimary01'
WHERE monitor_server = 'OddServer03'


Before running your own version of the commands above, which will try to clean out the system tables, it's worth considering whether you want to try skipping the GUI when trying to reset the Log Shipping configuration.  If you want to try that, I would recommend scripting out your current configuration (Right-click your database in SQL Server Management Studio --> Choose Properties --> Click Transaction Log Shipping in left sidebar -> Click Script Configuration button on the bottom right).  

You can then examine the results for what commands it uses as well as whether you see it inserting the OddServer03 value anywhere.  At that point, you could clean everything up as described in the 3 steps above and then add Log Shipping back via the script you extracted and updated as needed.  That would give you 100% control over what is actually executing and it would be less likely for the OddServer03 to somehow end up in the configuration.  



NOTE: If any of the above links break, use Google to search for them: - dba.stackexchange.com site for the title of 'Can I deactivate log shipping jobs without raising errors' - blog.sqlauthority.com site for the title of 'Removing Multiple Databases from Log Shipping'