Download SQL Server 2016 CTP3 Technical Overview

Exploring New Features in
SQL Server 2016
Triad SQL Server Users Group
Jessica M. Moss, Microsoft Data Platform MVP
jessica@jessicammoss.com
May 31, 2016
Agenda
Mission-critical performance
Performance
Operational analytics
Insights on operational data;
works with in-memory OLTP and
disk-based OLTP
In-memory OLTP
enhancements
Greater T-SQL surface area,
terabytes of memory supported,
and greater number of parallel
CPUs
Query data store
Security
Always encrypted
Enhanced AlwaysOn
Row-level security
Round-robin load balancing of
replicas
Sensitive data remains encrypted
at all times with ability to query
Apply fine-grained access control
to table rows
Other enhancements
Support for SSIS with AlwaysOn
Real-time obfuscation of data to
prevent unauthorized access
Native JSON
TDE support for storage of InMemory OLTP tables
Query data as points in time
Automatic failover based on
database health
DTC for transactional integrity
across database instances with
AlwaysOn
Audit success/failure of database
operations
Temporal database
support
Three synchronous replicas for
automatic failover across domains
Dynamic data masking
Monitor and optimize query plans
Expanded support for JSON data
Availability
Enhanced auditing for OLTP with
ability to track history of record
changes
Scalability
Enhanced database
caching
Cache data with automatic,
multiple TempDB files per instance
in multicore environments
Mission-critical performance
Performance
Operational analytics
Insights on operational data;
works with in-memory OLTP and
disk-based OLTP
In-memory OLTP
enhancements
Greater T-SQL surface area,
terabytes of memory supported,
and greater number of parallel
CPUs
Query data store
Security
Always encrypted
Enhanced AlwaysOn
Row-level security
Round robin load balancing of
replicas
Sensitive data remains encrypted
at all times with ability to query
Apply fine-grained access control
to table rows
Other enhancements
Support for SSIS with AlwaysOn
Real-time obfuscation of data to
prevent unauthorized access
Native JSON
TDE support for storage of inmemory OLTP tables
Query data as points in time
Automatic failover based on
database health
DTC for transactional integrity
across database instances with
AlwaysOn
Audit success/failure of database
operations
Temporal database
support
Three synchronous replicas for
auto failover across domains
Dynamic data masking
Monitor and optimize query plans
Expanded support for JSON data
Availability
Enhanced auditing for OLTP with
ability to track history of record
changes
Scalability
Enhanced database
caching
Cache data with automatic,
multiple TempDB files per instance
in multi-core environments
Operational analytics
The ability to run analytics queries concurrently with operational
workloads using the same schema
Not a replacement for:
• Extreme analytics performance queries possible only using customized schemas (e.g.
Star/Snowflake) and pre-aggregated cubes
• Data coming from non-relational sources
• Data coming from multiple relational sources requiring integrated analytics
Performance
Operational analytics with columnstore index
B-tree index
Key points
Create an updateable NCCI for analytics
queries
Delete bitmap
Drop all other indexes that were created
for analytics
Delta row groups
Columnstore index is maintained just like
any other index
Nonclustered columnstore index (NCCI)
Performance
No application changes
Query optimizer will choose columnstore
index where needed
Operational analytics: columnstore on in-memory tables
Hash index
No explicit delta row group
Rows (tail) not in columnstore stay in In-Memory OLTP table
No columnstore index overhead when operating on tail
Range index
Deleted Rows Table
In-Memory OLTP table
Tail
Background task migrates rows from tail to columnstore in
chunks of 1 million rows not changed in last 1 hour
Deleted Rows Table (DRT) – Tracks deleted rows
Columnstore data fully resident in memory
Persisted together with operational data
No application changes required
Performance
ALTER TABLE Sales.SalesOrderDetail
ALTER INDEX PK_SalesOrderID
REBUILD
WITH (BUCKET_COUNT=100000000)
ALTER support
Full schema change support: add/alter/drop
column/constraint
Add/drop index supported
Surface area improvements
T-SQL surface area: New
{LEFT|RIGHT} OUTER JOIN
Disjunction (OR, NOT)
UNION [ALL]
SELECT DISTINCT
Subqueries (EXISTS, IN, scalar)
Almost full T-SQL coverage including scaler user-defined
functions
Improved scaling
Increased size allowed for durable tables; more sockets
Other improvements
MARS support
Lightweight migration reports
Performance
Improved scaling
Other enhancements include:
7x
Performance
In-Memory OLTP
engine has been
enhanced to scale
linearly on servers up to
4 sockets
Improvements in Management Studio
Lightweight performance analysis
• Transaction Performance Analysis report pinpoints
hotspots in the application
Generating migration checklists
• Migration checklists show unsupported features used
in current disk-based tables and interpreted T-SQL
stored procedures
• Generated checklists for all or some tables and
procedures
• Use GUI or PowerShell
Performance
New Transaction Performance Analysis Overview report
New report replaces the
need to use the
Management Data
Warehouse to analyze
which tables and stored
procedures are candidates
for in-memory
optimization
Performance
Support for Transparent Data Encryption (TDE)
Windows Operating System
Level Data Protection
DPAPI encrypts the Service Master Key
SQL Server
Instance Level
Service Master Key
Created at a time of SQL Server setup
Service Master Key Encrypts the Database
master Key for the master Database
Master
Database Level
Database Encryption Key
Statement:
CREAT MASTER KEY…
Database Master Key of the master Database
creates a certificate in the master database
Statement:
CREATE CERTIFICATE…
The certificate encrypts the database
Encryption Key in the user database
User Database
Level
Database Encryption Key
Statement:
CREATE DATABASE ENCRYPTION KEY…
The entire user database is secured by the
Datbase Encryption Key (DEK) of the user
database by using transparent database
encryption
Statement:
ALTER DATABSE… SET ENCRYPTION
Performance
In SQL Server 2016, the
storage for memoryoptimized tables will be
encrypted as part of enabling
TDE on the database
Simply follow the same steps
as you would for a disk-based
database
Query Store
Your flight data recorder
for your database
Problems with query performance
Website
Is down
Temporary
perf issues
DB
upgraded
Performance
Database
is not
working
Impossible
to predict /
root cause
Regression
caused by
new bits
Fixing query plan choice regressions is difficult
•
Query plan cache is not well-suited for performance troubleshooting
Long time to detect the issue (TTD)
•
•
Which query is slow? Why is it slow?
What was the previous plan?
Long time to mitigate (TTM)
•
•
Can I modify the query?
How to use plan guide?
The solution: Query Store
Dedicated store for query workload performance data
Captures the history of plans for each query
Captures the performance of each plan over time
Persists the data to disk (works across restarts, upgrades, and recompiles)
Significantly reduces the time to diagnosis the problem
Find regressions and other issues in seconds
Allows you to force previous plans from history
DBA is now in control
Performance
Keeping stability while upgrading to SQL Sever 2016
SQL Server 2016
QO enhancements tied to database compatibility level
Install bits
Keep
existing
compat.
level
Performance
Run Query
Store
(create a
baseline)
Move to
vNext
CompatLevel
Fix
regressions
with plan
forcing
Monitoring performance by using the Query Store
The Query Store
feature provides DBAs
with insight on query
plan choice and
performance
Performance
Live query statistics
View CPU/memory usage, execution time, query
progress, and more
Enables rapid identification of potential
bottlenecks for troubleshooting query
performance issues
Allows drill down to live operator level statistics:
Number of generated rows
Elapsed time
Operator progress
Live warnings
Performance
JavaScript Object Notation (JSON) and Microsoft
Dynamics
Bing
OneDrive
Office
Yammer
TFS
A top feature request on MS Connect (1050 votes)
Performance
Use cases
Generate
web service
content
Flexible
database
schema
Analyze
JSON
documents
Service
integration
Generate JSON that will be
returned to clients
Make reasonable trade-offs
in database schema design
Parse, query, and analyze
JSON documents
Exchange information with
various services
Performance
JSON in SQL Server
Built-in functions
Table 2 JSON
Formats result set
as JSON text.
[
SO43659
2011-05-31T00:00:00
MSFT
59.99
1
SO43661
2011-06-01T00:00:00
Nokia
24.99
3
ISJSON
JSON_VALUE
JSON_MODIFY
{
},
{
JSON 2 table
Migrates JSON
text to table
Performance
]
}
"Number":"SO43659",
"Date":"2011-05-31T00:00:00"
"AccountNumber":"AW29825",
"Price":59.99,
"Quantity":1
"Number":"SO43661",
"Date":"2011-06-01T00:00:00“
"AccountNumber":"AW73565“,
"Price":24.99,
"Quantity":3
Main facts
No custom type or index
Store as NVARCHAR
Does JSON work with X?
Does NVARCHAR work with X?
Yes, with in-memory, row-level security, stretch, compression, encryption, and more
Yes, with all client drivers
Different from DocumentDB
Table to JSON
JSON output:
SO43659
2011-05-31T00:00:00
MSFT
59.99
1
SO43661
2011-06-01T00:00:00
Nokia
24.99
3
SELECT
Number AS [Order.Number], Date AS [Order.Date],
Customer AS Account,
Price AS 'Item.UnitPrice', Quantity AS 'Item.Qty'
FROM SalesOrder
FOR JSON PATH
Performance
JSON to Table
@json:
SELECT
*
FROM OPENJSON (@json)
WITH (
)
Performance
Number
Date
Customer
Quantity
varchar(200)
datetime
varchar(200)
int
N'$.Order.Number',
N'$.Order.Date',
N'$.Account',
N'$.Item.Quantity'
SO43659
2011-05-31T00:00:00
Microsoft
1
SO43661
2011-06-01T00:00:00
Nokia
3
Built-in functions
CREATE TABLE SalesOrderRecord (
Id int PRIMARY KEY IDENTITY,
OrderNumber NVARCHAR(25) NOT NULL,
OrderDate DATETIME NOT NULL,
JSalesOrderDetails NVARCHAR(4000)
CONSTRAINT SalesOrderDetails_IS_JSON
CHECK ( ISJSON(JSalesOrderDetails)>0 ),
JSON is plain text
ISJSON guarantees
consistency
Quantity AS
CAST(JSON_VALUE(JSalesOrderDetails, '$.Order.Qty') AS int)
)
GO
CREATE INDEX idxJson
ON SalesOrderRecord(Quantity)
INCLUDE (Price);
Optimize further with
computed column and
INDEX
Demonstration: JSON
Temporal tables: Query back in time
Data changes over time
Tracking and analyzing changes is often important
Temporal in DB
Time travel
Data audit
Automatically tracks history of data changes
Enables easy querying of historical data states
Advantages over workarounds
Slowly changing
dimensions
Performance
Repair record-level
corruptions
Simplifies app development and maintenance
Efficiently handles complex logic in DB engine
How to start with temporal
ANSI 2011
compliant
No change in programming model
CREATE temporal
TABLE PERIOD FOR
SYSTEM_TIME…
ALTER regular_table
TABLE ADD
PERIOD…
DML
SELECT * FROM temporal
INSERT / BULK INSERT
UPDATE
DELETE
DDL
Performance
MERGE
Querying
New Insights
Temporal
Querying
FOR SYSTEM_TIME
AS OF
FROM..TO
BETWEEN..AND
CONTAINED IN
Temporal database support: BETWEEN
SELECT * FROM
Person.BusinessEntityContact
FOR SYSTEM_TIME BETWEEN @Start AND @End
WHERE ContactTypeID = 17
Provides correct information
about stored facts at any
point in time, or between
two points in time
There are two orthogonal sets of scenarios with
regards to temporal data:
System (transaction)-time
Application-time
Performance
Demonstration: temporal
Mission-critical performance
Performance
Operational analytics
Insights on operational data;
Works with in-memory OLTP and
disk-based OLTP
In-memory OLTP
enhancements
Greater T-SQL surface area,
terabytes of memory supported,
and greater number of parallel
CPUs
Query data store
Security
Always encrypted
Enhanced AlwaysOn
Row-level security
Round robin load balancing of
replicas
Sensitive data remains encrypted
at all times with ability to query
Apply fine-grained access control
to table rows
Other enhancements
Support for SSIS with AlwaysOn
Real-time obfuscation of data to
prevent unauthorized access
Native JSON
TDE support for storage of InMemory OLTP tables
Query data as points in time
Automatic failover based on
database health
DTC for transactional integrity
across database instances with
AlwaysOn
Audit success/failure of database
operations
Temporal database
support
Three synchronous replicas for
auto failover across domains
Dynamic data masking
Monitor and optimize query plans
Expanded support for JSON data
Availability
Enhanced auditing for OLTP with
ability to track history of record
changes
Scalability
Enhanced database
caching
Cache data with automatic,
multiple TempDB files per instance
in multi-core environments
The need for Always Encrypted
Prevents data
disclosure
Queries on
encrypted data
Application
transparency
Client-side encryption of
sensitive data using keys that
are never given to the
database system
Support for equality
comparison, including join,
group by, and distinct
operators
Minimal application changes
via server and client library
enhancements
Allows customers to securely store sensitive data outside of their trust boundary.
Data remains protected from high-privileged, yet unauthorized users.
Security
Types of encryption for Always Encrypted
Randomized encryption
Encrypt('123-45-6789') = 0x17cfd50a
Repeat: Encrypt('123-45-6789') = 0x9b1fcf32
Allows for transparent retrieval of encrypted
data but NO operations
More secure
Deterministic encryption
Encrypt('123-45-6789') = 0x85a55d3f
Repeat: Encrypt('123-45-6789') = 0x85a55d3f
Allows for transparent retrieval of encrypted
data AND equality comparison
E.g. in WHERE clauses and joins, distinct,
group by
Security
Two types of encryption
available
Randomized encryption uses a method
that encrypts data in a less predictable
manner
Deterministic encryption uses a method
which always generates the same
encrypted value for any given plaintext
value
The need for row-level security
Protect data privacy by ensuring
the right access across rows
Customer 1
Fine-grained access control over specific rows in a database
table
Customer 2
Help prevent unauthorized access when multiple users
share the same tables, or to implement connection filtering
in multitenant applications
SQL Database
Administer via SQL Server Management Studio or SQL
Server Data Tools
Enforcement logic inside the database and schema is
bound to the table
Security
Customer 3
Benefits of row-level security (RLS)
Fine-grained
access control
Application
transparency
Centralized
security logic
Keeping multitenant
databases secure by limiting
access by other users who
share the same tables
RLS works transparently at
query time, no app changes
needed
Enforcement logic resides
inside database and is
schema-bound to the table it
protects providing greater
security. Reduced application
maintenance and complexity
Compatible with RLS in other
leading products
Store data intended for many consumers in a single database/table while at the same time
restricting row-level read and write access based on users’ execution context.
Security
Common RLS use cases
Traditional RLS workloads
Custom business logic to determine which rows each user can SELECT, INSERT, UPDATE, and DELETE
based on their role, department, and security level
Target sectors: Finance, insurance, healthcare, energy, and government
Multitenant databases
Ensuring tenants can only access their own rows of data in a shared database, with enforcement logic
in the database rather than in the app tier
For example: multitenant shards with elastic database tools in SQL Database
Reporting, analytics, and data warehousing
Different users access same database through various reporting tools, and work with different
subsets of data based on their identity/role
Security
Dynamic data masking
Prevent the abuse of sensitive
data by hiding it from users
Table.CreditCardNo
4465-6571-7868-5796
4468-7746-3848-1978
Configuration made easy in the new Azure portal
Policy-driven at the table and column level, for a
defined set of users
Data masking applied in real-time to query
results based on policy
Multiple masking functions available (e.g. full,
partial) for various sensitive data categories
(credit card numbers, SSN, etc.)
Security
4484-5434-6858-6550
SQL Database
SQL Server 2016
Real-time data masking;
partial masking
Benefits of dynamic data masking
Regulatory
compliance
Sensitive data
protection
Agility and
transparency
Data is masked on the fly, with
underlying data in the database
remaining intact. Transparent to
the application and applied
according to user privilege
Limit access to sensitive data by defining policies to obfuscate specific database fields, without
affecting the integrity of the database.
Security
How it works
Table.CreditCardNo
Limit sensitive data exposure by
obfuscating it to non-privileged users
On-the-fly obfuscation of data in query results
Policy-driven on the table and column
Multiple masking functions available for various sensitive data categories
Flexibility to define a set of privileged logins for un-masked data access
By default, database owner is unmasked
See: https://msdn.microsoft.com/en-us/library/mt130841.aspx
Security
4465-6571-7868-5796
4468-7746-3848-1978
4484-5434-6858-6550
Azure DB
Dynamic
masking
On-the-fly masking of
sensitive data in query results
Demonstration:
Dynamic Data Masking
Mission-critical performance
Performance
Operational analytics
Insights on operational data;
Works with in-memory OLTP and
disk-based OLTP
In-memory OLTP
enhancements
Greater T-SQL surface area,
terabytes of memory supported,
and greater number of parallel
CPUs
Query data store
Security
Always encrypted
Sensitive data remains encrypted
at all times with ability to query
Row-level security
Apply fine-grained access control
to table rows
Round-robin load balancing of
replicas
Automatic failover based on
database health
Other enhancements
Support for SSIS with AlwaysOn
Real-time obfuscation of data to
prevent unauthorized access
Native JSON
TDE support for storage of inmemory OLTP tables
Query data as points in time
Three synchronous replicas for
automatic failover across domains
DTC for transactional integrity
across database instances with
AlwaysOn
Audit success/failure of database
operations
Temporal database
support
Enhanced AlwaysOn
Dynamic Data Masking
Monitor and optimize query plans
Expanded support for JSON data
Availability
Enhanced auditing for OLTP with
ability to track history of record
changes
Scalability
Enhanced database
caching
Cache data with automatic,
multiple TempDB files per instance
in multi-core environments
Enhanced AlwaysOn: The need
• Detects failures reliably
• Able to handle multiple failures
• Unified, simplified solution
• Easy to deploy, manage, and monitor
• Reuse existing investments
• SAN/DAS environments
• Able to use HA hardware resources
• Fast, transparent failover
Availability
Enhanced AlwaysOn Availability Groups
Greater scalability
Unified HA solution
Load balancing readable secondaries
Increased number of automatic failover targets
Log transport performance
AG_Listener
Improved manageability
AG
Hong Kong
DTC support
Asynchronous data (Secondary)
Movement
Database-level health monitoring
AG
AG
New York
(Primary)
Availability
Synchronous data
Movement
New Jersey
(Secondary)
Group Managed Service Account
Domain-independent Availability Groups
More than two automatic failover targets
Increased solution scale
Increased resiliency
Now any sync secondary can be a target for automatic failover
Total of three automatic failover targets
Availability
Log transport performance
New hardware pushes boundaries
Very high transaction rates have caused problems when coupled with
fast hardware (i.e. PCIe flash storage)
The entire pipeline has been reworked end to end
Result is much lower CPU consumption, and much better performance
Performance bar: comparison to the performance of a standalone server
Availability
Distributed Transaction Coordinator (DTC) support
Previously, any distributed transactions touching a database in
an availability group are not allowed
Many customers run unsupported, at risk to their data and reputation
Many enterprise applications need cross-database transactions
Fully supported in SQL Server 2016
Joint effort with Windows
Requires specific patch in order to work cleanly
Other requirements:
• Availability groups must be running on Windows Server 2016 Technical Preview 2
• Availability groups must be created with the CREATE AVAILABILITY GROUP command and the WITH
DTC_SUPPORT = PER_DB clause. You cannot currently alter an existing availability group
• Learn more: https://msdn.microsoft.com/en-us/library/ms366279.aspx
Availability
Database-level failover trigger
Previously, Availability Groups only monitor the health of the instance
A database can be offline or corrupt, but will not trigger a failover as long as the instance itself is
healthy
SQL Server 2016: option to also monitor the health of the databases in the
Availability Group
Databases going offline trigger a change in the health status
Availability
Domain-independent Availability Groups
New feature in Windows Server 2016
Environments supported:
Cross domains (with trust)
Cross domains (no trust)
No domain at all
Cluster management via PowerShell only
SQL management as normal
Uses certificate-secured endpoints like DBM
Availability
Mission-critical performance
Performance
Operational analytics
Insights on operational data;
Works with in-memory OLTP and
disk-based OLTP
In-memory OLTP
enhancements
Greater T-SQL surface area,
terabytes of memory supported,
and greater number of parallel
CPUs
Query data store
Security
Always encrypted
Enhanced AlwaysOn
Row-level security
Round robin load balancing of
replicas
Sensitive data remains encrypted
at all times with ability to query
Apply fine-grained access control
to table rows
Other enhancements
Support for SSIS with AlwaysOn
Real-time obfuscation of data to
prevent unauthorized access
Native JSON
TDE support for storage of inmemory OLTP tables
query data as points in time
Automatic failover based on
database health
DTC for transactional integrity
across database instances with
AlwaysOn
Audit success/failure of database
operations
Temporal database
support
Three synchronous replicas for
auto failover across domains
Dynamic Data Masking
Monitor and optimize query plans
Expanded support for JSON data
Availability
Enhanced auditing for OLTP with
ability to track history of record
changes
Scalability
Enhanced database
caching
Cache data with automatic,
multiple TempDB files per instance
in multicore environments
Scalability Improvements: Enhanced database caching
Supports caching data with automatic, multiple TempDB
files per instance in multicore environments
Reduces metadata and allocation contention for TempDB
workloads, improving performance and scalability
Scalability
Enhanced support for Windows Server
Hardware acceleration for TDE encryption/decryption
Parallelizing the built-in decryption function to improve
read performance
Results in dramatically better response times for queries with encrypted data columns
Scalability
Cloud Platform System advantages
Rack 1
Rack 2
Rack 3
Rack 4
Management
Switches
Switches
Switches
Networking
Networking
Networking
Networking
Edge
Edge
Edge
Compute
Compute
Compute
Compute
Storage
Storage
Storage
Storage
Edge
Scalability
Partnership with Dell
Combination of optimized
hardware/software
Designed specifically to reduce the
complexity and risk of implementing a
self-service cloud
CPS can go from delivery to live within
days—not months—and lets service
providers and enterprises move up the
stack to focus on delivering services to
users
Hyperscale cloud
Hybrid solutions
Stretch Database
Stretch operational tables in a secure manner
into Azure for cost-effective historic data
availability. Works with Always Encrypted and
row-level security
Power BI with on-premises data
New interactive query with Analysis Services.
Customer data stays behind your firewall
Enhanced backup to Azure
Faster restore times and 50% reduction in
storage. Supports larger DBs with block blobs
and custom backup schedule with local staging
Simplicity
Easy migration of on-premises SQL
Server
Simple point-and-click migration to Azure
Suite of advisors for upgrading to SQL
Server 2016
SQL Server 2016 Upgrade Advisor in the
adoption of new SQL Server features
Simplified Add Azure
Replica Wizard
Automatic listener configuration for AlwaysOn
in Azure VMs
Consistency
Common development,
management, and identity tools
Including Active Directory, Visual Studio, HyperV, and System Center
Consistent experience from SQL
Server on-premises to Microsoft
Azure IaaS and PaaS
Hyperscale cloud
Hybrid solutions
Stretch Database
Stretch operational tables in a secure manner
into Azure for cost-effective historic data
availability. Works with Always Encrypted and
row-level security
Power BI with on-premises data
New interactive query with Analysis Services.
Customer data stays behind your firewall
Enhanced backup to Azure
Faster restore times and 50% reduction in
storage. Supports larger DBs with block blobs
and custom backup schedule with local staging
Simplicity
Easy migration of on-premises SQL
Server
Simple point-and-click migration to Azure
Suite of advisors for upgrading to SQL
Server 2016
SQL Server 2016 Upgrade Advisor for the
adoption of new SQL Server features
Simplified Add Azure
Replica Wizard
Automatic listener configuration for AlwaysOn
in Azure VMs
Consistency
Common development,
management, and identity tools
Including Active Directory, Visual Studio, HyperV, and System Center
Consistent experience from SQL
Server on-premises to Microsoft
Azure IaaS and PaaS
Stretch Database: Ever-growing data, ever-shrinking IT
Massive tables (hundreds of
millions/billions of rows, TBs size)
Users want/need to retain data
indefinitely
Cold data infrequently accessed but
must be online
Datacenter consolidation
Maintenance challenges
Business SLAs at risk
Hybrid solutions
What to do?
Expand server and storage
Move data elsewhere
Delete
Stretch SQL Server into Azure
Securely stretch cold tables to Azure with remote query processing
Capability
Azure
Stretch large operational tables
from on-premises to Azure with
the ability to query
Benefits
SQL
SERVER
2016
Hybrid solutions
Stretch Database architecture
Local
data
Eligible
data
Internet Boundary
Local
database
Linked Servers
How it works
Creates a secure linked server
definition in the on-premises SQL
Server
Remote
endpoint
Remote
data
Azure
Hybrid solutions
Linked server definition has the
remote endpoint as the target
Provisions remote resources and
begins to migrate eligible data, if
migration is enabled
Queries against tables run against
both the local database and the
remote endpoint
Queries continue working
• Business applications continue working
without disruption
• DBA scripts and tools work as before.
All controls still held in local SQL Server
• Developers continue building or
enhancing applications with existing
tools and methods
Hybrid solutions
Utilize existing investments in Power BI
Power BI connectivity
Connect to on-premises SQL
Server Analysis Services models
Support for Tabular and multidimensional models
Hybrid solutions
Empower analysts to securely and easily access on-premises data
Anyone can refresh
With Personal Gateway, everyone can keep their data in Power BI synced with their on-premises sources
No IT dependency
Easy and simple to use
Personal Gateway is a lightweight one-click installer
Allows for seamless data refresh
Enables IT departments to centrally deploy and manage to a large
group of users
Centralized way to refresh content in Power BI
Configure, manage, and maintain a group of Enterprise Gateways
Transparently refresh data uploaded to Power BI for business users and analysts
Centralized access control
Centrally manage the set of users who can access the underlying data sources
Assign user permissions
BI governance
Empower IT to monitor and audit usage
Gain visibility into Enterprise Gateway usage, such as commonly accessed data
sources, and the users accessing them
Direct Access
Direct access to a wide range of data sources, including SQL Server
Hybrid solutions
Get more from existing Reporting Services investments
Monitor the most important metrics and
trends in one place
Pin on-premises SQL Server Reporting
Services report elements to Power BI
dashboards
Hybrid solutions
Enhanced backup to Azure
Managed backup
Backup to Azure block blobs
Granular control of the backup
schedule
Cost savings on storage
Azure Storage snapshot
backup
Significantly improved restore
performance
Fastest method for creating
backups and running restores
More granular control over Azure
Storage
Uses SQL Server database files on
Azure Blob storage
Local staging support for faster
recovery and resilient to transient
network issues
Support for system databases
Supports simple recovery mode
Hybrid solutions
Hyperscale cloud
Hybrid solutions
Stretch Database
Stretch operational tables in a secure manner
into Azure for cost-effective historic data
availability. Works with Always Encrypted and
row-level security
Power BI with on-premises data
New interactive query with Analysis Services.
Customer data stays behind your firewall
Enhanced backup to Azure
Faster restore times and 50% reduction in
storage. Supports larger DBs with block blobs
and custom backup schedule with local staging
Simplicity
Easy migration of on-premises SQL
Server
Simple point-and-click migration to Azure
Suite of advisors for upgrading to SQL
Server 2016
SQL Server 2016 Upgrade Advisor for the
adoption of new SQL Server features
Simplified Add Azure
Replica Wizard
Automatic listener configuration for AlwaysOn
in Azure VMs
Consistency
Common development,
management, and identity tools
Including Active Directory, Visual Studio, HyperV, and System Center
Consistent experience from SQL
Server on-premises to Microsoft
Azure IaaS and PaaS
Easily migrate databases and objects to Microsoft Azure
Simple single-click migration experience
User DB
System objects
SQL settings
Capability
Along with schema and data, migrate other
system objects (logins, jobs, and certificates)
Migrate SQL Server settings (trace flags,
default language, and memory settings)
Migration Wizard
Benefits
Literally as simple as point and click
On-premises
Simplicity
Microsoft Azure
Upgrade workflow today
Discover
Monitoring
Assess
Upgrade is a complicated process
with many considerations
Most upgrades are actually
“migration projects”
Wave of SQL 2005 upgrades coming
Review
Plan
Implement
Simplicity
Upgrade Advisor focused on the
“Assess” phase: find functional
blockers
Suite of advisors
Modern, scenario-driven user
experience
Suite of advisors
Rules built on DACFx
Modern telemetry and feedback
pipeline
Designed to move to a
community project
Upgrade Advisor and Stretch
Database initial scenarios
Independent install via WebPI or
download center
Auto-update notifications
Simplicity
Simplified Add Azure Replica Wizard
Automatic listener configuration
Previously, this requires manually
configuring the listener
SQL Server 2016
Simplicity
Simplified Add Azure Replica Wizard
Add Azure Replica Wizard adds a replica of your databases to Azure Blob Storage
Group listener is created and configured within the wizard
Clients can seamlessly connect to the Azure replica after failover, as soon as the
Wizard completes its setup and without additional complex steps
Simplicity
Hyperscale cloud
Hybrid solutions
Stretch Database
Stretch operational tables in a secure manner
into Azure for cost-effective historic data
availability. Works with Always Encrypted and
row-level security
Power BI with on-premises data
New interactive query with Analysis Services.
Customer data stays behind your firewall
Enhanced backup to Azure
Faster restore times and 50% reduction in
storage. Support larger DBs with block blobs
and custom backup schedule with local staging
Simplicity
Easy migration of on-premises SQL
Server
Simple point-and-click migration to Azure
Suite of advisors for upgrading to SQL
Server 2016
SQL Server 2016 Upgrade Advisor for the
adoption of new SQL Server features
Simplified Add Azure
Replica Wizard
Automatic listener configuration for AlwaysOn
in Azure VMs
Consistency
Common development,
management, and identity tools
Including Active Directory, Visual Studio, HyperV, and System Center
Consistent experience from SQL
Server on-premises to Microsoft
Azure IaaS and PaaS
Consistent tools and experience
Consistent and
integrated platform
Virtualization
Complete data platform
Unified management and DevOps
Flexible development paradigm
Common identity
Consistency
Consistent tools
Consistency across:
On-premises, private cloud, public cloud
SQL Server local, VM, SQL Database
Scalability, availability, security, identity, backup and restore, and replication
Plethora of data sources
Reporting, integration, processing, and analytics
All supports hybrid cloud
Consistency
SQL Server, Azure VMs, SQL Database
Shared
Lower Cost
Platform as
a service
Infrastructure
as a service
Virtual
SQL
SQL Database
Virtualized Databases
SQL Server on physical machines
SQL Server in on-premises VMs (private
cloud)
SQL Server in Azure VMs (public cloud)
SQL Server in Azure VM
Virtualized machines
SQL Database (public cloud)
SQL Server
Physical Machines(raw Iron)
Off Premises
Higher Administration
Consistency
SQL
SQL
Software as
a services
SQL Server Private Cloud
Virtualized Machines +
Appliances
Physical
Dedicated
Higher
Cost
Options:
Hybrid Cloud
On Premises
Lower Administration
Deeper insights across data
Access any data
PolyBase
Insights from data across SQL
Server and Hadoop with the
simplicity of T-SQL
Enhanced SSIS
Designer support for previous SSIS
versions
SSIS Improvements for Azure
services
Scale and manage
Enterprise-grade
Analysis Services
Enhanced performance and
scalability for Analysis Services
Single SSDT in Visual Studio
2015
Build richer analytics solutions as
part of your development projects
in Visual Studio
Enhanced MDS
Excel add-in 15x faster; more
granular security roles; archival
options for transaction logs; and
reuse entities across models
Powerful insights
Mobile BI
Business insights for your onpremises data through rich
visualization on mobile devices
with native apps for Windows, iOS,
and Android
Enhanced Reporting
Services
New modern reports with rich
visualizations
Advanced analytics
R integration
Bringing predictive analytic
capabilities to your relational
database
Expand your “R” script library with
Microsoft Azure Marketplace
Deeper insights across data
Access any data
PolyBase
Insights from data across SQL
Server and Hadoop with the
simplicity of T-SQL
Enhanced SSIS
Designer support for previous SSIS
versions
SSIS Improvements for Azure
services
Scale and manage
Enterprise-grade
Analysis Services
Enhanced performance and
scalability for analysis services
Single SSDT in Visual Studio
2015
Build richer analytics solutions as
part of your development projects
in Visual Studio
Enhanced MDS
Excel add-in 15x faster; more
granular security roles; archival
options for transaction logs; and
reuse entities across models
Access any data
Powerful Insights
Mobile BI
Business insights for your onpremises data through rich
visualization on mobile devices
with native apps for Windows, iOS
and Android
Enhanced Reporting
Services
New modern reports with rich
visualizations
Advanced analytics
R integration (CTP3)
Bringing predictive analytic
capabilities to your relational
database
Analytics libraries (CTP3)
Expand your “R” script library with
Microsoft Azure Marketplace
PolyBase for SQL Server 2016:
Interest in big data spurs customer demand
Increase in number and
variety of data sources
that generate large
quantities of data
Realization that data is
“too valuable” to delete
Dramatic decline in the
cost of hardware,
especially storage
$
Adoption of big data technologies like Hadoop
PolyBase and queries
Provides a scalable, T-SQL-compatible query processing
framework for combining data from both universes
Access any data
PolyBase View in SQL Server 2016
PolyBase View
• Execute T-SQL queries against
relational data in SQL Server and
‘semi-structured’ data in HDFS
and/or Azure
• Leverage existing T-SQL skills and
BI tools to gain insights from
different data stores
• Expand the reach of SQL Server to
Hadoop(HDFS)
Access any data
PolyBase use cases
Access any data
SSIS improvements for SQL Server 2016
AlwaysOn support
Incremental deployment of
packages
SSIS
DB
New York
(Primary)
AlwaysOn
Availability Groups
Secondary for
SSISDB
Improved project
upgrade
Improved project upgrade support
SSIS
DB
New Jersey
(Secondary)
Error column name support
Custom log level
Package template
OData V4 support
Designer improvements
SQL Server
Server2012
2016
SSIS
ProjectXX
SSIS Project
Access any data
One designer multi-version support
AlwaysOn support for SSISDB
High availability for the SSIS catalog
DBA can set up AlwaysOn availability groups
for the SSIS Catalog
Access any data
Deploy packages to Integration Services server
private static void Main(string[] args)
{
// Connection string to SSISDB
var connectionString = "Data Source=.;Initial Catalog=SSISDB;Integrated
Security=True;MultipleActiveResultSets=false";
using (var sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
var sqlCommand = new SqlCommand
{
Connection = sqlConnection,
CommandType = CommandType.StoredProcedure,
CommandText = "[catalog].[deploy_packages]"
};
var packageData = Encoding.UTF8.GetBytes(File.ReadAllText(@"C:\Test\Package.dtsx"));
// DataTable: name is the package name without extension and package_data is byte
array of package.
var packageTable = new DataTable();
packageTable.Columns.Add("name", typeof(string));
packageTable.Columns.Add("package_data", typeof(byte[]));
packageTable.Rows.Add("Package", packageData);
// Set the destination project and folder which is named Folder and Project.
sqlCommand.Parameters.Add(new SqlParameter("@folder_name", SqlDbType.NVarChar,
ParameterDirection.Input, "Folder", -1));
sqlCommand.Parameters.Add(new SqlParameter("@project_name", SqlDbType.NVarChar,
ParameterDirection.Input, "Project", -1));
sqlCommand.Parameters.Add(new SqlParameter("@packages_table", SqlDbType.Structured,
ParameterDirection.Input, packageTable, -1));
var result = sqlCommand.Parameters.Add("RetVal", SqlDbType.Int);
result.Direction = ParameterDirection.ReturnValue;
}
}
sqlCommand.ExecuteNonQuery();
Access any data
Deployment options
Integration Services Deployment Wizard
SQL Server Management Studio
Deploy packages stored procedure
Object model API
Enhancements
SQL Server Data Tools for Business
Intelligence
Error column name support
No need to manually determine the column name
Developer can see the error column name in
both the data viewer and editor
Developer can also see the IntToString
lineage ID mapping in the log
Developer can also programmatically get the
column name using the lineage ID
Custom Log level
Customize your SSIS log with exactly what you need
Developer can create and use a customized
log level other than the default
Access any data
Package template
Reuse code as templates
Developer can save part of the package as a
template and reuse it in the design of other
packages
Access any data
OData Source support for V4 protocol
Access any data
SSIS improvements for Azure services
Capability
SQL
Stretch large operational tables
from on-premises to Azure with
the ability to query
Benefits
Access any data
Demonstration: SSIS
Deeper insights across data
Access any data
PolyBase
Insights from data across SQL
Server and Hadoop with simplicity
of T-SQL
Enhanced SSIS
Designer support for previous SSIS
versions
SSIS Improvements for Azure
services
Scale and manage
Enterprise-grade
Analysis Services
Enhanced performance and
scalability for Analysis Services
Single SSDT in Visual Studio
2015
Build richer analytics solutions as
part of your development projects
in Visual Studio
Enhanced MDS
Excel add-in 15x faster; more
granular security roles; archival
options for transaction logs; and
reuse entities across models
Powerful Insights
Mobile BI
Business insights for your onpremises data through rich
visualization on mobile devices
with native apps for Windows, iOS
and Android
Enhanced Reporting
Services
New modern reports with rich
visualizations
Advanced analytics
R integration (CTP3)
Bringing predictive analytic
capabilities to your relational
database
Expand your “R” script library with
Microsoft Azure Marketplace
Analysis Services themes for SQL Server 2016
Improved productivity and performance
Increased
productivity
Scale and manage
SQL Server Analysis Services
Scale and manage
Multidimensional in SQL Server 2016
Scale and manage
Dev/Ops in SQL Server 2016
Improved manageability and development
Scale and manage
Custom authentication in SQL Server 2016
New security feature for tabular and MOLAP
EUN=Effectiveusername
Admin
ActAs=AnyString
Admin
SQL Server
Analysis Services
CustomData
Scale and manage
Tabular models in SQL Server 2016
New rich modeling capabilities
DimOrganization
OrganizationKey
ParnentOrganization
PercentageOfOwn…
OrganizationName
CurrencyKey
DimDepartmentGr…
DimDepartmentGroupKey…
ParentDepartmentGr…
Improved Direct Query
DepartmentGroupNa…
FactFinance
FinanceKey
Performance enhancements
DateKey
OrganizationKey
DepartmentGroup…
ScenarioKey
DimAccount
AccountKey
ParenAccountKey
AccountCodeAlter…
ParenAccountCod…
DimAccount
ScenarioKey
ScenarioName
Scale and manage
AccountDescription
Dev/Ops
Tabular performance in 2016
New rich modeling capabilities
Bidirectional cross-filtering
translations
Display folders
Drill-through
Scale and manage
Tabular model in 2016
New DAX functions including:
DATEDIFF
SELECTCOLUMNS
PERCENTILE(X)
SUMMARIZECOLUMNS
CONCATENATE(X)
GROUPBY
MEDIAN(X)
UNION
GEOMEAN(X)
CROSSFILTER
PRODUCT(X)
VARIABLE support
XIRR
XNPV
Scale and manage
Direct Query
Analyze relational and non-relational data directly
Power BI
designer
APS
Excel
SQL Server
Scale and manage
Unified Visual Studio Data Tools: Rich BI platform
Develop and deliver BI solutions faster and at lower costs
Capability
SSDT-BI and SSMS for Visual Studio 2015
Rich data modeling enhancements
New DAX analytical functions
MDX Query Plan tool for performance optimizations and troubleshooting
{JSON}
Scale and manage
Benefits
Master Data Services: Areas of improvement
Scale and manage
Performance and scale
Massive improvements to
performance and scalability
15x
Scale and manage
Modeling and management
Modeling and
management
Transaction log retention
Configurable settings for retaining the MDS transaction history
table to enable automatic truncation
Improved hierarchy management
Simplified hierarchy types
Easily find and manage unused members
Display name for each object
More control over the names displayed for a given object, including
Code and Name attributes
Scale and manage
Security and administration
Granular security permissions
Allows permissions to be set around read, write, create, and
delete
Multiple administrator roles
Support for Super User and Model Admin roles allows for
multiple system administrators, and model level administrators
Scale and manage
MDS improvements
Business rules and hierarchy
management from Excel
Use Excel add-in to manage business rules, such as
creating and editing business rules
Many-to-many relationships in
derived hierarchies
Model many-to-many relationships between two entities
by mapping with a third entity
Merge conflicts
Perform merge conflicts and republish the changes
Scale and manage
Deeper insights across data
Access any data
PolyBase
Insights from data across SQL
Server and Hadoop with simplicity
of T-SQL
Enhanced SSIS
Designer support for previous SSIS
versions
SSIS improvements for Azure
services
Scale and manage
Enterprise-grade
Analysis Services
Enhanced performance and
scalability for Analysis Services
Single SSDT in Visual Studio
2015
Build richer analytics solutions as
part of your development projects
in Visual Studio
Enhanced MDS
Excel add-in 15x faster; more
granular security roles; archival
options for transaction logs; and
reuse entities across models
Powerful insights
Mobile BI
Business insights for your onpremises data through rich
visualization on mobile devices
with native apps for Windows, iOS,
and Android
Enhanced Reporting
Services
New modern reports with rich
visualizations
Advanced analytics
R integration (CTP3)
Bringing predictive analytic
capabilities to your relational
database
Expand your R script library with
Microsoft Azure Marketplace
Mobile BI apps for SQL Server (previously Datazen)
On-premises implementations are
optimized for SQL Server
Rich, interactive data visualization on
all major mobile platforms
No additional cost for customers of
SQL Server Enterprise Edition 2008 or
later and Software Assurance
View on any major mobile platform
Access reports with online/offline
support
Powerful insights
Mobile Reports Publisher
Data visualization and publishing
Powerful insights
Mobile BI architecture overview
Internet boundary
Enterprise environment
Authentication
Data sources
SQL Server
Publisher
app
Datazen Enterprise
Server
Viewer
apps
Analysis Services
File
Web
browser
Powerful insights
Authentication mode
Default mode
Active Directory
Active Directory Federation
Services (ADFS)
External
http://www.datazen.com/docs/?article=server/authentication
Powerful insights
Adding a custom data provider
Use the control panel to
export the schema file
Add schema file to your
local server
Sample files are available for
download after today’s
session
Powerful insights
SQL Server 2016 Reporting Services and mobile reporting
CREATE
Datazen Publisher
MANAGE
SQL Server 2016
Reporting Services
(Native mode)
 Datazen Server now a
component of SSRS
SQL Server
Report Builder
Report Designer in
SQL Server Data Tools
CONSUME
Datazen Windows app
Datazen phone app
Datazen Android app
Datazen iOS app
Report Server web portal
(paginated and mobile reports)
SSRS reports rendered as PDF
SQL Server 2016
Reporting Services
(SharePoint integrated mode)
Pin paginated report elements
to Power BI dashboards
SharePoint web
Powerful insights
Enterprise reporting with modern design
Modernizing
CONSUME
Use modern design tools
Create reports based on more enterprise data
sources
Modern paginated reports
Mobile reports
Other
data sources
Powerful insights
Simplifying
Datazen integration for mobile reports
Modern reports
Modern Report Builder and Manager look and
feel
Visual Studio 2015 support
Powerful insights
Consume anywhere, anytime
Find, view, and manage through modern web
portal
Support for modern HTML5 browsers
Printing without plug-ins
PowerPoint rendering and export
Powerful insights
Demonstration: SSRS
Deeper insights across data
Access any data
PolyBase
Insights from data across SQL
Server and Hadoop with simplicity
of T-SQL
Enhanced SSIS
Designer support for previous SSIS
versions
SSIS improvements for Azure
services
Scale and manage
Enterprise-grade
Analysis Services
Enhanced performance and
scalability for Analysis Services
Single SSDT in Visual Studio
2015
Build richer analytics solutions as
part of your development projects
in Visual Studio
Enhanced MDS
Excel add-in 15x faster; more
granular security roles; archival
options for transaction logs; and
reuse entities across models
Powerful insights
Mobile BI
Business insights for your onpremises data through rich
visualization on mobile devices
with native apps for Windows, iOS,
and Android
Enhanced Reporting
Services
New modern reports with rich
visualizations
Advanced analytics
R integration
Bringing predictive analytic
capabilities to your relational
database
Expand your R script library with
Microsoft Azure Marketplace
What is R?
Most widely used data analysis software
Used by more than 2 million data scientists, statisticians, and analysts
Most powerful statistical programming language
Flexible, extensible, and comprehensive for productivity
Create beautiful and unique data visualizations
As seen in The New York Times, The Economist, and FlowingData
Thriving open-source community
Leading edge of analytics research
Fills the talent gap
New graduates prefer R
Machine learning examples
Credit card fraud detection
Digit recognition for handwriting recognition
Real-time facial and image recognition engines
Recommendation engines for products in e-commerce
Stock trading
Medical diagnosis of diseases and lifespan
Crash prediction of machines in the cloud
R plotting
•
•
•
•
•
•
•
•
http://homes.cs.washington.edu/~jheer//files/zoo/?utm_source\x3dtwitterfeed\x26utm_medium\x3dtwitter
Revolution R Enterprise and SQL
Big data analytics platform
Based on open source R
High-performance, scalable, full-featured
Statistical and machine-learning algorithms are
performant, scalable, and distributable
Write once, deploy anywhere
Scripts and models can be executed on a variety of
platforms, including non-Microsoft (Hadoop,
Teradata in-DB)
Integration with the R Ecosystem
Analytic algorithms accessed via R function with similar
syntax for R users. Arbitrary R functions/packages can be
used in conjunction
Advanced analytics
SQL Server 2016 R integration scenario
Exploration
Use RRE from R IDE to analyze large datasets
and build predictive and embedded models
with the compute happening on the SQL
Server machine (SQL Server compute
context)
Operationalization
Developer can operationalize R script/model
over SQL Server data by using T-SQL
constructs
DBA can manage resource, secure, and
govern R runtime execution in SQL Server
Advanced analytics
R script usage from SQL Server
Original R script:
Calling R script from SQL Server:
IrisPredict <- function(data, model){
library(e1071)
predicted_species <- predict(model, data)
return(predicted_species)
}
/* Input table schema */
create table Iris_Data (name varchar(100), length int, width int);
/* Model table schema */
create table my_iris_model (model varbinary(max));
library(RODBC)
conn <- odbcConnect("MySqlAzure", uid = myUser, pwd =
myPassword);
Iris_data <-sqlFetch(conn, "Iris_Data");
Iris_model <-sqlQuery(conn, "select model from my_iris_model");
IrisPredict (Iris_data, model);
declare @iris_model varbinary(max) = (select model from
my_iris_model);
exec sp_execute_external_script
@language = 'R'
, @script = '
IrisPredict <- function(data, model){
library(e1071)
predicted_species <- predict(model, data)
return(predicted_species)
}
IrisPredict(input_data_1, model);
'
, @parallel = default
, @input_data_1 = N'select * from Iris_Data'
, @params = N'@model varbinary(max)'
, @model = @iris_model
with result sets ((name varchar(100), length int, width int
, species varchar(30)));
The values highlighted in yellow are the SQL queries embedded in the original R script
The values highlighted in aqua are the R variables that bind to SQL variables by name
Advanced analytics
R script library in Microsoft Azure Marketplace
Extensibility
Example solutions
Fraud detection
R Integration
Launch
External
Process
Sales forecasting
Warehouse efficiency
R
Benefits
New R
scripts
010010
100100
010101
010010
100100
010101
010010
100100
010101
Microsoft Azure
Machine Learning Marketplace
Predictive maintenance
Analytic library
T-SQL interface
010010
100100
010101
010010
100100
010101
Relational data
Built into SQL Server
Advanced analytics
Data Scientist
Interacts directly
with data
Data Developer/DBA
Manages data and
analytics together
Faster deployment of ML models
Faster performance
(moves compute close to the
data)
Improved scalability
Benefits
The Microsoft
data platform
Apps
Reports
Dashboards
Ask
Mobile
Microsoft Azure
Orchestration
Extract, transform,
load
Information
management
Prediction

Relational
Non-relational
Analytical
Streaming
Internal and
external
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on
the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.