Trains.com

Amtrak nationwide service meltdown starting early March 24 till Mar 26 1200 CDT

2431 views
23 replies
1 rating 2 rating 3 rating 4 rating 5 rating
  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Amtrak nationwide service meltdown starting early March 24 till Mar 26 1200 CDT
Posted by blue streak 1 on Saturday, March 25, 2023 5:45 AM

For some reason Amtrakhas been unable to connect to the PTC server for many trains especially CHI.  This started early AM Friday and is continuing as of 0645 Saturday morning.  Some long distance originators yesterday cancelled  =  2, 3, 7, 21.   All other LD departures from CHI in service distuption but running except 59 normal.  West coast and east coast OK but almost all took origination delays.  Almost all regionals out of CHI cancelled and 171 out of ROA delayed.

It is too early to know eactly how all this started and if freight RRs have had any problems.  If this is an Amtrak problem some ITs need to be ????.  Here is link to ongoing problems.

Amtrak Alerts (@AmtrakAlerts) / Twitter

1679671941109.png

  • Member since
    March 2016
  • From: Burbank IL (near Clearing)
  • 13,480 posts
Posted by CSSHEGEWISCH on Saturday, March 25, 2023 9:52 AM

I'm not sure if this is related but my morning ride to work (Metra Southwest) was stopped at South Branch Bridge for about 5-10 minutes shortly after 7:00AM.  The crew announced that it was due to PTC problems.

The daily commute is part of everyday life but I get two rides a day out of it. Paul
  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Saturday, March 25, 2023 11:10 AM

As of 1200 EDT almost all CHI regionals in and out cancelled.  #s 1 & 22 cancelled at NOL & SAS.  What LD at CHI are going to be cancelled this afternoon?

  • Member since
    January 2002
  • From: Canterlot
  • 9,518 posts
Posted by zugmann on Saturday, March 25, 2023 3:31 PM

blue streak 1
It is too early to know eactly how all this started and if freight RRs have had any problems.  If this is an Amtrak problem some ITs need to be ????.  Here is link to ongoing problems.

Has affected frieght RRs, too. Sounds like the I-ETMS system was giving fits.

I don't even know how much of that is Amtrak, vs WABTEC vs. Rockwell Collins or whomever is involved.  

  

The opinions expressed here represent my own and not those of my employer, any other railroad, company, or person.

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Saturday, March 25, 2023 5:01 PM

All Hias cancelled today.  But Amtrak shines for one item.  ACSES is working without a hick up.  All NEC trains running OK with slowing  NYP <> NRO work.  beginning to wonder if system hack?   As of 1800 EDT 3, 4, 5, 21 all ready cancelled out of CHI,

EDIT: 4 has been restored/

Amtrak Advisory | Server Issues Causing Delays

 

Due to a rail-industry control system that continues to experience intermittent communications issues, some Amtrak trains to and from Chicago have been canceled through late afternoon. We're working around-the-clock with our partners to continue troubleshooting and addressing these issues.

Affected customers have been notified and we are working to resolve this as soon as possible. Updates will be provided as available.

You may change your reservation via the Amtrak app or Amtrak.com. If your trip has been canceled, cancellation fees will be waived. Follow @AmtrakAlerts on Twitter for real-time service information and updates.

 

 

 
 

  

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Saturday, March 25, 2023 10:33 PM

Now official  30, 48, 50, 59 cancelled.  That makes it 100% LD trains cancelled out of CHI for today Satursay.  Is Gardner & Harris happy ?.  FRA needs to change PTC initiation protocols.

That brings up the question .  Where is the FRA and Sec of Transportation??

  • Member since
    January 2002
  • From: Canterlot
  • 9,518 posts
Posted by zugmann on Saturday, March 25, 2023 10:46 PM

blue streak 1
  FRA needs to change PTC initiation protocols.

So you want them to now nullify the billions of dollars of technology that was mandated ?

 

  

The opinions expressed here represent my own and not those of my employer, any other railroad, company, or person.

  • Member since
    May 2003
  • From: US
  • 24,937 posts
Posted by BaltACD on Saturday, March 25, 2023 10:57 PM

blue streak 1
Now official  30, 48, 50, 59 cancelled.  That makes it 100% LD trains cancelled out of CHI for today Satursay.  Is Gardner & Harris happy ?.  FRA needs to change PTC initiation protocols.

That brings up the question .  Where is the FRA and Sec of Transportation??

Bull Hockey!  You overlooked AutoTrain in both directions.

Never too old to have a happy childhood!

              

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Saturday, March 25, 2023 11:08 PM

Balt you are correct both Auto trains cancelled on Saturday.  That is an almost all CSX operation except Sun Rail and Amtrak going north from SFA.  I had just looked at CHI but we have 1, 2, 22 also  cancelled.   Have no idea if revelant but our CSX track only had one northbound all day and it was slow.

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Saturday, March 25, 2023 11:30 PM

zugmann

 

 
blue streak 1
  FRA needs to change PTC initiation protocols.

 

So you want them to now nullify the billions of dollars of technology that was mandated ?

 

Yes for emergency situations.  Depending on how much the freight RRs are affecred the class 1s will get new excuses for delivery delays.

  • Member since
    May 2003
  • From: US
  • 24,937 posts
Posted by BaltACD on Sunday, March 26, 2023 8:33 AM

My question is - Who's Servers were down the prevented the initial testing and linking of PTC?  Amtrak?  Freight Carriers?

Why would Amtrak not use the servers of the Freight Carriers who are actually operating their trains?

The problem as I understand it - PTC is required to be linked and operative at a train's origin.  Once underway, PTC can fail and be 'cut out' that will allow a train to continue to its destination.

Never too old to have a happy childhood!

              

  • Member since
    January 2002
  • From: Canterlot
  • 9,518 posts
Posted by zugmann on Sunday, March 26, 2023 12:05 PM

blue streak 1
Yes for emergency situations.  Depending on how much the freight RRs are affecred the class 1s will get new excuses for delivery delays.

I don't know if this would qualify as an "emergency". 

  

The opinions expressed here represent my own and not those of my employer, any other railroad, company, or person.

  • Member since
    February 2016
  • From: Texas
  • 1,537 posts
Posted by PJS1 on Sunday, March 26, 2023 12:37 PM

BaltACD
  Why would Amtrak not use the servers of the Freight Carriers who are actually operating their trains? 

Why do we have Amtrak?  Creating it was a mistake. 
 
If passenger trains are critical for the well being of the United States, the government should have subsidized those trains that were being operated by the investor owned railroads.
 
What we have is another Washington, D.C. bureaucracy with more than 20,000 employees guided by executives and managers that seem to have little idea what they are doing. 

Rio Grande Valley, CFI,CFII

  • Member since
    May 2003
  • From: US
  • 24,937 posts
Posted by BaltACD on Sunday, March 26, 2023 12:55 PM

PJS1
 
BaltACD
  Why would Amtrak not use the servers of the Freight Carriers who are actually operating their trains?  
Why do we have Amtrak?  Creating it was a mistake. 
 
If passenger trains are critical for the well being of the United States, the government should have subsidized those trains that were being operated by the investor owned railroads.
 
What have is another Washington, D.C. bureaucracy with more than 20,000 employees guided by executives and managers that seem to have little idea what they are doing. 

Politics of half a century ago and today are different - just as life from half a century ago is different from today.

Remember, when Amtrak was created it was expected to be dead in five years.

Never too old to have a happy childhood!

              

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Sunday, March 26, 2023 2:04 PM

As  of 1800 EDT 3, 5, 7, 21 All out CHI on time.  s well 339, 393 also on time

Eagle  #21 & CZ #5 got out of CHI on time.  335 HIA also out both after 1300 CDT.  Maybe problem fixed?   However, with all the equipment out of position probably 6  days to  get all midwest LD trains running normally.?   Amtrak managemet where are you ?? 

  • Member since
    January 2002
  • From: Canterlot
  • 9,518 posts
Posted by zugmann on Sunday, March 26, 2023 3:13 PM

Is it even possible to use another road's servers?  

 

 

  

The opinions expressed here represent my own and not those of my employer, any other railroad, company, or person.

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Sunday, March 26, 2023 5:23 PM

How how long will it take to get everything straightened out?  Note:  Nothing on any Amtrak system publications had notice saying that boh Auto Train cancellations happened SAT.  Only checking reservation's Status check showed them cancelled.  How were the riders notified? What the heck?

  • Member since
    May 2003
  • From: US
  • 24,937 posts
Posted by BaltACD on Sunday, March 26, 2023 6:05 PM

zugmann
Is it even possible to use another road's servers?  

And when we are saying 'Servers' what data is actually involved?

When I was still working (Pre-PTC) on CSX we had continual issues getting timely and accurate train data from Amtrak - appropriate data was Train ID, Engine(s) number of cars, Conductor & Engineer names and their ID numbers as well as the On Duty times that apply to each as well as their HOS time.  Pre-PTC we could inquire of the crew and apply the information manually.  What additional data is now required with the implementation of PTC, I don't know.

Never too old to have a happy childhood!

              

  • Member since
    December 2007
  • From: Georgia USA SW of Atlanta
  • 11,831 posts
Posted by blue streak 1 on Sunday, March 26, 2023 8:02 PM

30 and 59 On time out of CHI.  However the planned late departure for 59 of 18:35 was met but  appears that it has stopped at the service yard for over 90 minutes so far. Now only 48 to go.

  • Member since
    March 2016
  • From: Burbank IL (near Clearing)
  • 13,480 posts
Posted by CSSHEGEWISCH on Monday, March 27, 2023 10:00 AM

BaltACD

 

 
PJS1
 
BaltACD
  Why would Amtrak not use the servers of the Freight Carriers who are actually operating their trains?  
Why do we have Amtrak?  Creating it was a mistake. 
 
If passenger trains are critical for the well being of the United States, the government should have subsidized those trains that were being operated by the investor owned railroads.
 
What have is another Washington, D.C. bureaucracy with more than 20,000 employees guided by executives and managers that seem to have little idea what they are doing. 

 

Politics of half a century ago and today are different - just as life from half a century ago is different from today.

Remember, when Amtrak was created it was expected to be dead in five years.

 
That was one of the reasons that Roger Lewis was appointed as Amtrak's first president.  A political favor could be repaid with no real consequences foreseen.
The daily commute is part of everyday life but I get two rides a day out of it. Paul
  • Member since
    September 2014
  • 1 posts
Posted by ANDREW S KIRK on Monday, March 27, 2023 2:30 PM

On Jan. 11 an FAA computer outage had grounded over 11,000 flights. Between last Thursday and Sunday, back office server issues with Amtrak I-ETMS (PTC) resulted in over 50 Amtrak train cancelations out of Chicago alone along with numerous spot cancelations. The FRA had a preliminary report the next day - we haven't seen anything from the FRA or Amtrak even offering to investigate. At least 15,000 and perhaps over 20,000 passengers suddenly had to find alternate means of reaching their destinations. It appears that neither Amtrak nor the company that offers the server service had backup facilities. Continuous tranportation can be provided if the carrier is willing to accommodate some redundancy in its facilities. Recently when CSX had a situation that affected the computers used by train dispatchers in the systemwide facility in Jacksonville, the dispatchers were moved into a back-up facilitiy until the main office was up and running. When I worked for Conrail, their busy Selkirk Yard kept running without a hitch after an October heavy/wet snow felled so many branches that the greater Albany, NY area lost commerial power for several days. Why? Because Selkirk had backup generators. I firmy believe that everyone who had a ticket who was denied transportation during this period needs to contact his/her elected officials. And

  • Member since
    January 2002
  • From: Canterlot
  • 9,518 posts
Posted by zugmann on Monday, March 27, 2023 4:28 PM

ANDREW S KIRK
I firmy believe that everyone who had a ticket who was denied transportation during this period needs to contact his/her elected officials. And

What's that going to do?  Really? 

It's because of the law that the trains couldn't deaprt without having PTC initiated.  They helped caused this mess. 

  

The opinions expressed here represent my own and not those of my employer, any other railroad, company, or person.

  • Member since
    July 2008
  • 2,323 posts
Posted by rdamon on Monday, March 27, 2023 4:40 PM

"We are currently experiencing a communication issue impacting the nationwide network managed by a third party. This is causing train cancellations and delays in some areas," Amtrak said in a statement. "Some trains may also be running at reduced speeds. Updates will be provided as available."

Didn't we have a thread about out-sourcing?

  • Member since
    May 2003
  • From: US
  • 24,937 posts
Posted by BaltACD on Monday, March 27, 2023 4:54 PM

ANDREW S KIRK
On Jan. 11 an FAA computer outage had grounded over 11,000 flights. Between last Thursday and Sunday, back office server issues with Amtrak I-ETMS (PTC) resulted in over 50 Amtrak train cancelations out of Chicago alone along with numerous spot cancelations. The FRA had a preliminary report the next day - we haven't seen anything from the FRA or Amtrak even offering to investigate. At least 15,000 and perhaps over 20,000 passengers suddenly had to find alternate means of reaching their destinations. It appears that neither Amtrak nor the company that offers the server service had backup facilities. Continuous tranportation can be provided if the carrier is willing to accommodate some redundancy in its facilities. Recently when CSX had a situation that affected the computers used by train dispatchers in the systemwide facility in Jacksonville, the dispatchers were moved into a back-up facilitiy until the main office was up and running. When I worked for Conrail, their busy Selkirk Yard kept running without a hitch after an October heavy/wet snow felled so many branches that the greater Albany, NY area lost commerial power for several days. Why? Because Selkirk had backup generators. I firmy believe that everyone who had a ticket who was denied transportation during this period needs to contact his/her elected officials. And

When traffic control systems malfunction or crash, operations cannot continue safely - it makes little difference if the mode of transportation is air, marine or railroad - all of which ARE NOT line of sight operations when operated at their customary maximum speeds.

I have been an airline passenger inroute and ON my connecting flight when the ATC lost all flight plans that the carriers had filed for flights to leave Atlanta.  Instead of a before Noon arrival at destination it was a after 8 PM arrival.  Doo Doo Happens.

In the early 2000's the CSX Mainframe computer system contracted a 'virus' or some other bug that 'bussied up' the comm lines between the Mainframe and the CADS for the dispatching system to the point the CADS would crash everytime it was rebooted and brought back on line.  Problem started about 0100 and by 0600 CSX was shut down - Dispatchers could not safely identify or move trains.  About Noon the busied comm lines was diagnosed as the problem, but the cause of that was still not diagnosed.  The comm lines were 'cut' and CADS was restored to operation about 1300.  With CADS being 'UP' the consist and crew information that was supplied from the Mainframe had to be sourced and entered manually by the Train Dispatchers.  The Train Bulletins for each crew going On Duty could not be sent over printers as that information had to pass from the CADS to the mainframe to go to the printers in the field, thus all Train Bulletins had to be 'hand faxed' to the appropriate locations - account CSX Rules, upon receipt hand faxed bulletins had to be 'read back' to confirm proper receipt of all data.  All available Extra Train Dispatchers were called in to work the read backs.  After 48 hours the real cause was determined and fixed and operations returned to normal.

I recently heard of a maritime collision in a port area because the ports traffic control system had a failure.

Traffic Control Systems in commercial transportation are not toys.

Never too old to have a happy childhood!

              

Join our Community!

Our community is FREE to join. To participate you must either login or register for an account.

Search the Community

Newsletter Sign-Up

By signing up you may also receive occasional reader surveys and special offers from Trains magazine.Please view our privacy policy