Book excerpt: With the increasing popularity of cloud computing service, IT companies are building and expanding their datacenters nationwide or worldwide. With hundreds of thousands of servers, a commercial datacenter consumes many mega-watts of power annually, and imposes significant electricity costs to its operator. In this dissertation, we focus on reducing energy cost for Internet-scale datacenters (IDCs) to promote more efficient datacenter operation. To reduce IDC energy consumption, load-aware capacity provisioning schemes have been considered where capacity can be reduced (by turning off servers or scheduling a low CPU frequency) when its load is lower. To reduce IDC energy cost, electricity price-aware load shifting is considered, where IDCs with a lower price/cost serve more load. Our work is related to the load-aware capacity provisioning and electricity price-aware load shifting. However, different from most existing work, we consider practical traffic and load models. We first consider traffic dynamics at each IDC. We consider both large time scale and small time scale traffic variation. We further explicitly differentiate the load demand. We consider both delay sensitive jobs which have strict quality of service requirements, and delay tolerant jobs. We propose joint capacity allocation and load shifting schemes for distributed IDCs with dynamic and differentiated load demand. Our work is summarized as follows. In datacenters, traffic demand varies both in large and small time scales. A datacenter with dynamic traffic often needs toover-provision active servers to meet the peak load demand, which incurs significant energy cost. In the dissertation, our first goal is to achieve an optimal tradeoff between energy efficiency and service performance over a set of distributed IDCs with dynamic demand. In particular, we consider the overload probability as the QoS metric, where overload is defined as service demand exceeding the capacity of an IDC. We require the overload probability at each IDC to be smaller than some predefined threshold. Our goal is thus to minimize total energy cost over all IDCs, subject to the overload constraint. We achieve the goal by dynamically adjusting server capacity and performing load shifting in different time scales. We propose three different load-shifting and joint capacity allocation schemes with different complexity and performance. Our load shifting schemes leverage electricity-price diversity. Meanwhile, they reduce the traffic burstiness such that a smaller capacity is required to satisfy the overload probability constraint. Thus, our schemes reduce energy consumption/cost even when all IDCs have the same electricity price. We use both simulated load traces and real traffic traces to evaluate the performance of the proposed schemes. Our results show that our proposed schemes are efficient in reducing energy cost, and robust in QoS provisioning. Our first work uses load-aware capacity provisioning schemes, where servers are turned on/off according to the load. Similar to most existing work, we only consider instantaneous (Internet) requests, which are explicitly or implicitly assumed to be delay-sensitive. On the other hand, in datacenters, there are many delay-tolerant jobs, such as background/maintainance jobs. In our second work, we explicitly differentiate delay-sensitive jobs and delay tolerant jobs. We focus on the problem of using delay-tolerant jobs to fill the extra capacity of datacenters, referred to as trough/valley filling. Giving a higher priority to delay-sensitive jobs, our schemes improve most existing demand-proportional resource provisioning schemes. Our goal is then to design intelligent trough filling mechanisms that are energy efficient and also achieve good delay performance. Specifically, we propose two joint dynamic speed scaling and traffic shifting schemes, one subgradient-based and the other queue-based. Our schemes assume little statistical information about the system, which is usually difficult to obtain in practice. In both schemes, energy cost saving comes from dynamic speed scaling, statistical multiplexing, electricity price diversity, and service efficiency diversity. In addition, good delay performance is achieved in the queue-based scheme via load shifting and capacity allocation based on queue conditions. Practical issues that may arise in datacenter networks are considered, including capacity and bandwidth constraints, service agility constraints, and load shifting cost. We use both simulated and real datacenter traces to evaluate the proposed schemes. Last, we also propose joint capacity provisioning and load shifting schemes for mixed traffic with delay sensitive and delay tolerant jobs. In these schemes, we introduce a delay constraint as the QoS requirement for delay sensitive jobs. We employ a priority queue to derive the capacity demand for delay sensitive jobs under given delay constraints, and further consider bandwidth and capacity sharing for delay tolerant jobs. We follow similar approaches in the second work to reduce energy cost by delay sensitive jobs and delay tolerant jobs, while guaranteeing the delay requirement for delay sensitive jobs and good delay performance for delay tolerant jobs.