Computer Room Shutdown this Weekend (1/22/2022-1/24/2022)

An Important Upcoming Shutdown Effecting MIT App Inventor

MIT App Inventor relies on services running both on campus and in
several cloud computing providers (Google, Amazon and IBM).

We distribute our services based on various factors, including
cost. Our least costly services are those which we run on campus.

This coming weekend (January 22, 2022 - January 24, 2022) the computer
room that we use on campus will be shutdown for some power
infrastructure work.

Systems in that computer room that are not connected to an
uninterruptible power supply (UPS) will be shutdown. Those connected
to a UPS may stay up. Which is to say the official word is that they
should stay up, but when I asked about a confidence level, the answer
I received was “we have never done this before.”

Anticipating this shutdown, we will be moving critical services
provided from this computer room into the cloud. Some services that
are not critical and are difficult to move easily will remain down for
the weekend.

The services that will not be available are mostly those needed by the
developers of MIT App Inventor. People who use MIT App Inventor
should not be effected. Report any issue in our Community
Forum
, which will not be
effected and will be operational.

The Buildservers

One of the important services we run from on campus are our “Build
Servers.” This is a set of largish systems, 21 in all, each with 8 CPU
cores, that are used when folks instruct MIT App Inventor to package
an Android application. Each buildserver can process 8 simultaneous
builds.

On Friday, we will move this service to AWS. However, instead of 21
servers, we will only operate 9 both because it is a weekend and
because large servers in AWS are not cheap. Of these 9, 3 will be
normal instances and 6 will be what AWS calls “spot” instances. Spot
instances are available at a significant discount, but are subject to
being shutdown by AWS if the resources they are using are needed for
regular instances (aka, people paying more!). We have done this in the
past and have not had an issue. However, there is a risk that some or
all of these spot instances may be shutdown, which will leave us with
only 3 running buildservers, which may be enough to limp along, but
some people may get the “buildserver busy” message when they attempt
to package an app. We will be monitoring the situation throughout the
weekend and will restart any spot instances that are
shutdown. However, there are times when we will be sleeping, so
monitoring may not be as close during the late evening and early
morning on the east coast of the United States (GMT-5).

12 Likes

A UPS or Uninterruptible power supply provides power supply only for some minutes, so that either the data center staff has the time to shut the device down in a regular way or the emergency power station delivers enough power. In case of a shut down of the power in a data center, the servers will shut down within some minutes.
see here: Uninterruptible power supply - Wikipedia

2 Likes

this topic should be pinned globally until computer room is back to normal

Hi, I just want to know that do we need to pay from now on to use the buildservers?

no...
Taifun

1 Like

No, you do not have to pay to use the buildservers. We do :slight_smile: .

The servers on UPS power have a separate power panel that is served by the utility (power company) and is backed up by a generator.

Normal power is in a separate power panel. That is the power panel that will be shutdown. The UPS providing power panel is not supposed to be touched. So, servers on the UPS should stay up.

Having said that, it is possible that the electricians will shut off both panels in which case we will lose the UPS backed servers (and possibly in an unclean shutdown if the computer room staff isn't aware that they would do that)

If we lose the UPS backed servers, there will be some disruption until I can bring up replacements in the cloud. In particular, the server we use to send mail is a server on UPS power. If it goes down, I will have to bring one up on the cloud (we have an account in the cloud that is allowed to send mail [aka can contact port 25] most providers block sending email from a cloud server). But it will take some time to get that up and running.

Note: Email sent from this community site is routed through our e-mail server.

-Jeff

2 Likes

The computer room work is complete, and all services should be back to their pre-work configuration.

-Jeff

4 Likes

now you may unpin this topic... :slight_smile: