PSA for failed token requests

[NOYB]

Failed again last night.

If I knew, or had confidence, that changing the time to a different one-hour window would work in the long term, I'd do it in a heartbeat. As it is, it's a crap shoot.

I've been updating during not peak time ever since started using epg123 and SD. Works fine. Test by doing a manual update during the time you'd like to move to for a few days to see how it goes.

[stuartm]

Sadly my times were off-peak AFAICT but still are failing. Am going to try making them even more off-peak, but the key tskeaway is they have been working for years at the current times and only recently started failing so something must have changed WRT to SD.

[garyan2]

I agree something changed at SD, but they haven't found anything yet. It appears that the high traffic times exacerbate whatever it may be and is causing folks to fail to retrieve tokens. Changing update times will give users a higher probability of success and is not a bad idea in general. What worked in the past without issue doesn't reflect the reality of today. Once root cause is determined and fix action applied, then everything should go back to "normal."

So what I have poorly been trying to communicate is, if your update failed, you can...
1. do nothing and continue to fail
2. trigger an update manually sometime during the off-peak hours until this clears
3. change your update time to an off-peak time

@stuartm: is your error that it couldn't get a token, or something else? Not everything can be blamed on the token issue.

[stuartm]

Interestingly last night one system got this:
[7/10/2021 2:32:00 AM] Beginning epg123 update execution. version 1.6.4.4
[7/10/2021 2:32:00 AM] ===============================================================================
[7/10/2021 2:32:00 AM] [ INFO] Beginning EPG123 update execution. 2021-07-10 08:32:00Z
[7/10/2021 2:32:00 AM] [ INFO] DaysToDownload: 16 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: True , OADOverride : False , TMDbCoverArt: True , IncludeSDLogos : False , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[7/10/2021 2:32:06 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[7/10/2021 2:32:06 AM] [ERROR] Failed to complete request. Exiting
[7/10/2021 2:32:06 AM] [ERROR] Did not receive a response from Schedules Direct for a token request.
[7/10/2021 2:32:06 AM] [ERROR] Failed to retrieve token from Schedules Direct. message: Did not receive a response from Schedules Direct for a token request.
[7/10/2021 2:32:06 AM] [ERROR] Failed to create MXF file. Exiting.

And the other system got this:
[7/10/2021 3:11:00 AM] Beginning epg123 update execution. version 1.6.4.4
[7/10/2021 3:11:00 AM] ===============================================================================
[7/10/2021 3:11:00 AM] [ INFO] Beginning EPG123 update execution. 2021-07-10 09:11:00Z
[7/10/2021 3:11:00 AM] [ INFO] DaysToDownload: 16 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: True , OADOverride : False , TMDbCoverArt: True , IncludeSDLogos : False , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[7/10/2021 3:11:03 AM] [ INFO] Token request successful. serverID: 20141201.web.debug , datetime: 2021-07-10T09:11:02Z
[7/10/2021 3:12:11 AM] [ INFO] Status request successful. account expires: 2022-06-02T18:25:38Z , lineups: 2/4 , lastDataUpdate: 2021-07-10T00:54:25Z
[7/10/2021 3:12:11 AM] [ INFO] system status: Online , message: No known issues.
[7/10/2021 3:12:11 AM] [ INFO] GetClientVersion() Unknown exception thrown. Message: The requested security protocol is not supported.
[7/10/2021 3:12:13 AM] [ INFO] Successfully retrieved TMDb configurations.
[7/10/2021 3:12:32 AM] [ INFO] Successfully requested listing of subscribed lineups from Schedules Direct.
[7/10/2021 3:12:32 AM] Entering BuildLineupServices() for 2 lineups.
[7/10/2021 3:13:15 AM] [ INFO] Successfully retrieved the station mapping for lineup USA-CO65363-X.
[7/10/2021 3:14:48 AM] [ INFO] Successfully retrieved the station mapping for lineup USA-OTA-80503.
[7/10/2021 3:14:48 AM] Exiting BuildLineupServices(). SUCCESS.
[7/10/2021 3:14:48 AM] Entering GetAllScheduleEntryMd5s() for 16 days on 675 stations.
[7/10/2021 3:19:48 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[7/10/2021 3:23:58 AM] [ INFO] Successfully retrieved Md5s for 312 station's daily schedules. (0:00:09:09.6981655 / 531.146 KB)
[7/10/2021 3:28:58 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[7/10/2021 3:33:58 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[7/10/2021 3:33:58 AM] [ERROR] Failed to complete request. Exiting
[7/10/2021 3:33:58 AM] [ERROR] Did not receive a response from Schedules Direct for 312 station's daily schedules. (0:00:10:00.0394540)
[7/10/2021 3:33:58 AM] [ERROR] Problem occurred during GetMd5ScheduleEntries(). Exiting.

[garyan2]

So the first machine timed out on trying to get a token request after 3 seconds. I'm guessing that nearly another 3 seconds was taken just to see what the latest epg123 version is from the SD servers.

The second machine managed to get a token, but then failed after 2 attempts with 5 minute timeouts each to download the initial schedule listings for your stations.

Either way, both are being executed during the peak hours, and both timed-out and failed to create a MXF file.

[garyan2]

The "GetClientVersion() Unknown exception thrown. Message: The requested security protocol is not supported" is weird. Not sure what that is from, but it is moot with the latest version of epg123.

[stuartm]

When I chose those times the advice at the time was avoid on the hour updates so I skewed them away from that. Obviously the definition of peak hours is now more stringent. Is there a graph or some such showing SD activity to aid in choosing appropriate update times?

[StinkyImp]

Is there a graph or some such --- to aid in choosing appropriate update times?

I'm considering using a clock face as a dartboard. Seems as good a system as any.

[garyan2]

I was going to ask SD to provide something to illustrate traffic on the API throughout the day, could be unitless to not give away any membership numbers. Robert gave the times of 12AM EST to 3AM PST as a ballpark when they have the most traffic.

[d00zah]

Is there a graph or some such --- to aid in choosing appropriate update times?

I'm considering using a clock face as a dartboard. Seems as good a system as any.

I have it on good authority that's how the weather is forecast.

[stuartm]

Of course here in MST that range pretty much covers nearly the whole night However, my 4:13 am system has been failing far less often. The problem is exacerbated since the SO is a morning person and I am a night owl so if trying to prevent updates while WMC is possibly in use leaves me with a narrower window.

[garyan2]

You don't need to prevent updates while WMC is in use. EPG123 will ensure that guide listings will not be imported while a recording is active. I haven't had any problems with an import occurring while watching TV.

My 4:22am MST updates have never failed yet during these last couple weeks as well as my 7:20am development system.

[StinkyImp]

My 4:22am MST updates have never failed yet during these last couple weeks...

I could probably go with the 4:00AM - 5:00AM MST window BUT now that it's been mentioned here there will be others that will want to flock to those times.

Maybe we can start a group schedule based on SD's traffic patterns and reserve our times?

Incorporate that with the clock face dartboard (and weather predictor) and I think we may have a winner!

[Space]

I was going to ask SD to provide something to illustrate traffic on the API throughout the day, could be unitless to not give away any membership numbers. Robert gave the times of 12AM EST to 3AM PST as a ballpark when they have the most traffic.

FYI: Without mixing time zones that's from midnight ET until 6am ET.

[StinkyImp]

FYI: Without mixing time zones that's from midnight ET until 6am ET.

Time Zone Conversions

[NOYB]

I just what you all to avoid using my update time slot. Probably the best way to do that is not to share it. Or to share a bogus time slot. Yeah that's it. Hey everyone high noon is the time slot to chose. Make it so.

My 4:22am MST updates have never failed yet during these last couple weeks...

I could probably go with the 4:00AM - 5:00AM MST window BUT now that it's been mentioned here there will be others that will want to flock to those times.

[StinkyImp]

I agree something changed at SD, but they haven't found anything yet. It appears that the high traffic times exacerbate whatever it may be and is causing folks to fail to retrieve tokens. Changing update times will give users a higher probability of success and is not a bad idea in general. What worked in the past without issue doesn't reflect the reality of today. Once root cause is determined and fix action applied, then everything should go back to "normal."

So what I have poorly been trying to communicate is, if your update failed, you can...
1. do nothing and continue to fail
2. trigger an update manually sometime during the off-peak hours until this clears
3. change your update time to an off-peak time

- whistling a peaceful tune -

Eight days and patiently waitin`, waitin`, waitin`...

Code: Select all

[7/4/2021 12:07:04 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/5/2021 12:07:04 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/6/2021 12:07:03 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/7/2021 12:07:02 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/8/2021 12:07:04 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/9/2021 12:07:03 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/10/2021 12:07:04 AM] [ERROR] Failed to retrieve token from Schedules Direct. 
[7/11/2021 12:07:01 AM] [ERROR] Failed to retrieve token from Schedules Direct.

I find myself reminiscing about the demise of Microsoft's EPG. Are we coming full circle??

[Gripweed]

On the other hand, this is from my log -

[7/8/2021 5:37:57 AM] [ERROR] Did not receive a response from Schedules Direct for a token request

That's it for token errors. I've been reading here since about the 4th of July and scratching my head about what all the hubbub is about. And I do one SD update in the morning (like above) and one in the afternoon (just before Primetime).

I won a turkey once for Thanksgiving Day about 40 years ago, otherwise I'm usually not that lucky.

[NOYB]

I do one SD update in the morning (like above) and one in the afternoon (just before Primetime).

Are there many schedule changes within a few hours of air time?

[Gripweed]

Let me put it to you this way, morning or before Primetime, I'm always getting something. Whether the afternoon updates have to do with that night's broadcasts is hard to say. Sometimes the VerifyLoad section of the log will show a last-minute show replacement for that night but it doesn't happen every night. Some days I seem to get more updates in the morning and other days it is the afternoon. Either way I get something new whether it is show scheduling/metadata, movie lookups, or artwork. For me, I think having two updates a day is worth it. I don't think there is any benefit to doing more than that.