Document code Product name Target readers Engineers Product version Edited by Optical Network technical support department Document version T2000 Maintenance and Operation Guide Drafted by: Date: Reviewed by: Date: Reviewed by: Date: Approved by: Date: T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Huawei Technologies Co, Ltd All Rights Reserved 2005-04-14 Huawei Confidential No disclosure without permission Page of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Revision Records Date 2005-04-14 Revised version Description Huawei Confidential No disclosure without permission Author Page of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Contents Chapter Purpose Chapter NMS Maintenance Guide Chapter NMS Operation Guide 2005-04-14 Huawei Confidential No disclosure without permission 12 Page of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Keyword: OptiX Optical Transmission NMS Routine Maintenance and Operation Guide Summary: OptiX Optical Transmission NMS Routine Maintenance and Operation Guide include: Purpose, NMS Maintenance, NMS Operation Abbreviation list: None Reference list: None 2005-04-14 Huawei Confidential No disclosure without permission Page of 16 T2000 Maintenance and Operation Guide Χηαπτερ Confidentiality Level: Internal Use Only Purpose This guide provides instructions on two operations: 、 Routine NMS maintenance for higher efficiency 、NMS operations for higher security and transmission system stability 1.1 Maintenance Purpose Guide the maintenance personnel to perform routine NMS maintenance: Perform specific preventive maintenance, find out and solve problems timely to ensure the normal running of the NMS 1.2 Operation Purpose Guide the maintenance personnel to operate the NMS in a more reasonable and proper way: Master basic NMS operations to improve operation security and avoid optical transmission system abnormality due to improper operations 1.3 Scope of Application This guide applies to the iManager T2000 NMS widely used in the networks In addition, it is also appropriate for the NES/RMS NMS The maintenance of dual-system and the iManager T2100 NMS is beyond the scope of this guide For more details, access the following Website: http://support.huawei.com Χηαπτερ NMS Maintenance Guide This chapter provides a guide to the maintenance of the NMS itself, including NMS hardware, operating system, NMS database and NMS software For the routine maintenance of optical transmission devices, refer to their maintenance guides 2005-04-14 Huawei Confidential No disclosure without permission Page of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only 2.1 Types of Routine NMS Maintenance Daily maintenance Daily maintenance refers to the maintenance performed every day, which keeps the maintenance personnel timely informed of the NMS operation On the basis of such information, they can find out problems and troubles timely for quick solutions Periodic maintenance Periodic maintenance is to maintain the system periodically Emergency maintenance Emergency maintenance addresses the maintenance tasks brought about by troubleshooting and network adjustment 2.2 Basic Principles for Routine Maintenance 1) Routinize the maintenance 2) Detail the maintenance plan and content for effective implementation 2.3 Routine NMS Maintenance Items The routine maintenance of the OptiX NMS includes the following items 2.3.1 Item Checking the NMS Hardware Checking the NMS hardware Cause None Operation Make sure that the hardware devices such as the NM workstation and terminal router run normally: Check the mouse, keyboard, monitor and printer If there are abnormalities, contact the hardware suppliers (e.g., SUN and DELL) immediately for solution Check the other devices such as routers (if configured) periodically 2.3.2 Checking System Disk Space Occupation Item Checking the system disk space occupation Cause It is suggested that the disk space occupation should not exceed 90%, otherwise the NMS efficiency will be decreased Operation • On a workstation, use the df –k command to query the disk space occupation Make sure that the occupation of the directories “/”, “/T2000” and “/opt” is not higher than 90% If in excess of 90%, it is suggested that the unidentified super large rubbish files or directories such as the repetitive backed up data, useless files, directories and log information be cleared • On a PC, check the occupation of the operating system disk and the occupation of the SQL SERVER and T2000 NM directories 2005-04-14 Huawei Confidential No disclosure without permission Page of 16 T2000 Maintenance and Operation Guide 2.3.3 Confidentiality Level: Internal Use Only Ensuring Password and Authority Security Item Cause Operation Ensuring password and authority security None Set authorities for different levels of maintenance personnel and designate a special personnel to manage the password and modify it periodically The authority of the NMS user should be consistent with that of the NE user Make sure that the NM station (or PC) is for NM purpose only and isolated from external systems (principally) When it connects to an external system, security measures such as isolation are required Prohibit maintenanceunrelated operations and uses of maintenance-unrelated floppy disks and software in the NMS in order to fence off computer virus 2.3.4 Monitoring the Database Item Monitoring the database Cause The NMS is abnormal if the database is not started Operation • Follow these steps to check MSSQL7 on a PC: Check the icon at the bottom right corner of PC A green arrow indicates that the database is running normally, while a red arrow indicates that the database is not enabled and you need to click it manually and then select [Start] to enable the database • Execute the following commands in the workstation terminal window to check the Sybase database: T2000%su sybase $isql -Usa -P -ST2000DBServer 1>quit $isql -Usa -P -ST2000DBServer_back 1>quit $su t2000 T2000% The Sybase command line prompt 1> must be shown at the terminal, indicating the T2000DBServer has been successfully installed and can be automatically started 2.3.5 Item Cause Monitoring the T2000 Server Monitoring the T2000 server The four server processes of mdp, ss, toposvr and ems are crucial to guarantee the normal running of T2000 server If all the above four server processes are enabled and converge to be one process, the server is normally enabled If one process is missed, the server is not totally enabled If one process has two IDs, the server is repeatedly enabled Restart the server forcefully in the later two circumstances Operation • Follow these steps to check the T2000 server process on a PC: Check the PCS icon at the right bottom corner of PC Green indicates that the t2000 server process is normal, while red indicates that the t2000 server process is abnormal and you need to double click this icon to check which of the four servers of mdp, ss, toposvr and ems are not running (The T2000V1R001 version 2005-04-14 Huawei Confidential No disclosure without permission Page of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only has the alarm process besides the above four processes) • Execute the following commands in the workstation terminal window: T2000%cd /T2000/server T2000%showt2000server Normally, the result should be as follows (The process ID shall be unique): user is t2000 show ems process pid = 1140 show mdp process pid = 1136 show ss process pid = 1137 show toposvr process pid = 1138 Note: For the T2000V1R002D008 version, multiple ss processes will be shown without affecting the running of NMS 2.3.6 Backing up NMS Data Periodically Item Backing up NMS data periodically Cause NMS data can be recovered timely when NMS is abnormal due to NMS fault and hardware damage Back up data periodically (once per month in the initial running stage of NMS and once per quarter when the system can run steadily) It is suggested to back up data after the important data modification or before the NMS upgrading and in other storage devices such as tape drive, magneto-optical disk and U disk Operation In routine maintenance, it is suggested to back up data onto MO and the operations are as follows: Manual backup mode • Workstation T2000%cd /T2000/server/database T2000%backupmo.bat T2000%tar cvf mo_bak_XXXX.tar mo • PC Enter the C:\T2000\server\database directory Double click backupmo.bat Compress the mo directory under the C:\T2000\server\database and back up it to the backup directory You can also back up data manually on the NMS interface (T2000 V1R004 and its succeeding version supports the automatic backup) In the NMS interface, click [System]-[Database backup]-[Manual backup] Automatic backup T2000 V1R004 and its succeeding version supports the automatic backup In the NMS interface, click [System]-[Database backup]-[Automatic backup] Data restoration Initialize database, copy the backed up MO data in the /T2000/server/database directory and then execute restoremo.bat under this directory (For the T2000 V1R004 and its succeeding versions at the workstation, 2005-04-14 Huawei Confidential No disclosure without permission Page of 16 T2000 Maintenance and Operation Guide Item Confidentiality Level: Internal Use Only Backing up NMS data periodically execute restoremo.sh ) Use the data restoration operation only when necessary Note: The above backup modes only apply to the data backup and restoration of the same NMS version For the data upgrade or restoration between different NMS versions, you can only adopt the script-exporting mode: Export the configuration script of NMS in the case that the data at the NMS side and the NE side are consistent, and click the menu [Maintenance console] [Maintenance] - [Configuration data management] - [Script export/import] to export the “Global configuration file” and then back up the script In addition, you can back up data through the dump mode on the workstation and through the SQL SERVER out-of-box backup tool on PC As the backup data is quite large, it is suggested to back up only once during the trial office period 2.3.7 Checking Log Dump Setting Periodically Item Checking log dump setting periodically Cause An overlarge log may slow down the NMS operation Although T2000 has provided an automatic log dump function, you should check the log dump configuration periodically to prevent it from being modified Operation Enter the NMS menu and select [System] - [Data dump] to check if the periodic log dump or overflow log dump have been configured for the alarm event, abnormality and performance event If not, reconfigure them Usually the log dump configuration has been defaulted by the NMS 2.3.8 Checking the Network’s ECC Routing Item Checking the network’s ECC routing Cause Verify the connections between the NMS and the NEs Operation Enter [Maintenance console] - [Communication] - [NE ECC link management]to select an NE and perform as follows: Go through the ECC routes of the gateway NE to make sure that all the subnet NEs are covered by the target NE Analyze the NEs far from the ECC to see if they select the shortest ECC route 2.3.9 Checking Data Consistency between the NMS and the NEs Item Checking data consistency between the NMS and the NEs Cause Avoid NE data configuration by other NMSs, which causes data inconsistence between the NMS and the NE and the possible impact on the services Operations This check is suggested under normal network running Enter the NMS menu and select [Configuration] - [Data consistency checking configuration] to check the consistency of all the NEs on whole network one by one 2005-04-14 Huawei Confidential No disclosure without permission Page 10 of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only If the inconsistent NEs exist, locate the reason and then upload 2.4 Routine NMS Maintenance Item Table Date: Maintenance Month Year Maintenance item type Equipment Environment Maintenan Maintenance status External situations(Power supply Daily Normal Maintenance s ce cycle Remark personnel Abnormal system, UPS, fire, dust and lightning) Temperature (Normal Daily Normal Abnormal Daily Normal Abnormal Daily Good Bad Daily Normal 15~30°C) Humidity (Normally 40%~65%) Equipment room cleanness(good/bad) Checking virus on PC NMS Check Abnormal Checking NMS’ login to NE Daily Normal Abnormal Checking alarm and abnormality Daily Normal Abnormal Daily Normal Abnormal Daily Normal Abnormal on NMS Checking the database running state Monitoring T2000 server running status Checking the hardware status Weekly Normal Abnormal Weekly Normal Abnormal Monthly Modified Unmodified Monthly Normal Abnormal Monthly Normal Abnormal Monthly Finished Unfinished Quarterly Normal of NMS Checking the disk space occupation Controlling the security of password and authority Checking the log dump configuration Checking the data consistency between NMS side and NE side NMS Data backup Remote maintenance test Abnormal Note: The above tables only touch the NMS routine maintenance suggestions For the other device maintenance, refer to their corresponding maintenance proposals 2005-04-14 Huawei Confidential No disclosure without permission Page 11 of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Χηαπτερ NMS Operation Guide 3.1 NMS Operation Description This section gives some suggestions on the T2000 NMS operation, aiming to instruct the maintenance staff with their routine NMS operations for higher reasonability, efficiency and security 3.1.1 Closing the T2000 NMS and the OS Item Closing the T2000 NMS and the OS Cause Always avoid abnormal shutdown while T2000 is running The NMS keeps transaction when it is running, so abnormal closedown, either to the system or to the computer, may cause damage to the data, the system or the hardware and hence an NMS fault Unreasonable operations include: Computer power-off when the NMS is running Illegal power-off (especially for SUN workstation) Suggestion Closing methods: Select [System] - [Close Server] from the NMS menu to close the NMS server Power off the computer normally (after normally closing the NMS) Windows OS: Select [Run] – [Close Computer] Solaris OS: Run #/usr/sbin/shutdown –y –g0 –i5 to power off the computer Note: If you are logging into the T2000 NMS as a user of administrator’s level, be sure not to close the running server by selecting the “Close Server” operation (available for user) by mistake 3.1.2 Closing T2000 Prior to Modifying the NM Server’s System Time Item Closing T2000 prior to modifying the NM server’s system time Cause It is a limit in the system Modifying the system time may lead to T2000 time coordinate confusion, and hence failure to trigger the T2000 server’s refresh message As a result, these abnormalities may occur: failure to refresh interface color, failure to close alarm acknowledgement dialog box, no alarm tone and non-configured state remaining after configuration data uploading Suggestion Set the NM server’s system time correctly during installation To modify the time, you must log out and reboot the server after modification 3.1.3 Using a Special NE User for Each NMS for NE Login Item Using a special NE user for each NMS for NE login Cause Avoid conflict among login users on an NE, which may cause the NMS to fail monitoring the NE (for example, alarms cannot be timely reported) Suggestion The NE allocates a special NE user for each NMS and defines the right for the NE user in accordance with that of the NMS user To log on to an NE from an NMS, firstly make sure that the NMS is using an NE user not used by any other 2005-04-14 Huawei Confidential No disclosure without permission Page 12 of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only NMSs 3.1.4 Setting Alarm Properties Reasonably to Reduce Unnecessary Alarms Item Setting Alarm properties reasonably to reduce unnecessary alarms Cause In certain circumstances, alarms are reported in multitude, such as the alarms reported by NE frames frequently and transiently, and the mass alarms reported by a port on service not launched The large amount of alarms reported to the NMS affect both the operation and the maintenance efficiency of the NMS Suggestion From the NMS, you can set delay for some alarms report by the NEs frequently and transiently If an NE port not in use reports alarms frequently, you can set alarm reverse on it If an NE generates unnecessary alarms, you can screen it from the NMS Setting entry: [Alarm] – [Alarm Setting] – [NE Alarm Properties Setting] – Set alarm screening, reverse and delay 3.1.5 Setting Performance Properties Reasonably to Reduce Unnecessary Performance Events Item Setting performance reasonably to reduce unnecessary performance events Cause Setting to automatically report all the performance events of each of many NEs may cause mass performance data of 15m or 24h reported to the NMS This affects both the operation and the maintenance efficiency of the NMS Suggestion Note that the default setting for all the performance events is “Monitor, prohibit auto report” in the NMS Set it to “Auto report” only in special conditions Or you may set certain performances data to be reported to the NMS at certain interval as required, or allow only 24h performance data to be reported 3.1.6 Paying Attention to the Results of NMS Operations Item Paying attention to the results of NMS operations Cause Proceeding with the next operation while unaware of the result of the current operation may result in misoperation Suggestion Before running the next command, make sure that the correct result of the current command sent by the NMS is returned Set reasonable frequencies for the timing tasks in the NMS 3.1.7 Precautions for Mass Data Operations Item Cause Precautions for mass data operations A mass data operation may cause the NMS or any NEs to be busy and the subsequent NMS timeout or background busy error This wastes time and can probably lead to operation failure Suggestion Operate less NEs (less than is recommended) and boards For example: Set a large volume of alarm screening, checking and synchronization tasks (no more than 500 a time is recommended) Set performance enable/disable and board properties 2005-04-14 Huawei Confidential No disclosure without permission Page 13 of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Query a large mount of alarm screening View the details of many boards and the version information of many of NEs, query the properties of many NE boards Query, activate and deactivate a lot of services (no more than 500 service a time is suggested) Download NE data Browse, activate, deactivate and delete discrete services in multitude Note: If a batch operation fails, it is suggested that it be executed in several times to lower the data volume of each time 3.1.8 Suggestion of Single NE Download at a Time Item Suggestion of single NE download at a time Cause Download involves mass configuration data transmission downward, so operation on several NEs at a time may trigger alarms/performance events during the process and hence affect the ECC communication efficiency and finally cause a download failure suggestion 3.1.9 Perform download on a single NE at a time, and proceed with another one after the former operation succeeds Using Full Activation Only when Necessary Item Cause Using full activation only when necessary The full activation operation downloads service configurations to an NE once again For an NE with a large volume of service configurations, this operation involves mass data transmission and may leave the device at risk Suggestion 3.1.10 Use ”incremental” activation to replace “full” activation if possible Precautions for Multi-Terminal NMS Operations Item Precautions for multi-terminal NMS operations Cause T2000 adopts the C/S architecture to support multi-terminal operations Improper operations, however, always fail Due to the limit of the host capabilities in handling concurrent services and the NMS data protection mechanism, operations involving bulk data traffic between several terminals and a single NE are likely to cause performance degradation and timeout to the terminals and finally an operation failure Suggestion Avoid operations involving bulk data traffic between several terminals and a single NE Avoid the case where one of several terminals performs upload and consistency check on an NE while another terminal performs other operations on the same NE Avoid the case where one or several terminals perform script import/export operation on an NE while another terminal operates the same NE Avoid the case where one terminal deletes or initiates data while another terminal creates or queries the data This may cause operation failure to the other terminal Avoid concurrence of path function and any of the following single-NE operations on several terminals: a) Path search and single-NE upload, consistency check and deletion of NE or board b) Protected subnet search and single-NE upload, consistency check, deletion 2005-04-14 Huawei Confidential No disclosure without permission Page 14 of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only 3.1.11 Keeping Data Synchronous when Configuring a Service under Multi-NMS Environment Item Keeping data synchronous when configuring a service under multi-NMS environment Cause In a multi-NMS environment, the data of the NMSs cannot be synchronized automatically For example, if a service is configured in NMS A but is not uploaded to NMS B, configuring the service in NMS B will lead to unforeseable result Suggestion Limit the operation rights of the NMSs and allow service configuration in only one NMS The other NMSs must verify data synchronization before any service configuration 3.1.12 Keeping Data Synchronous after Configuration Modification through Command Lines Item Keeping data synchronous after configuration modification through command lines Cause Failure to refresh data in the NMS after configuration modification through command lines may cause unforeseable results to the follow on NMS operations Suggestion Generally, the command line configuration is not recommended, because most of the configuration tasks can be completed in the NMS After modifying any configuration, service or multiplex section parameter through command lines, you must refresh the NMS data immediately 3.1.13 Backing up NE Database Right after NE Configuration Change Item Backing up NE database right after NE configuration change Cause Failure to update the FDB right after configuration modification may have impact on the services in the case of power-off Suggestion 3.1.14 Back up the NE database with the NMS right after NE configuration modification/download Using Service-Impact Operations Only when Necessary Item Using service-impact operations only when necessary Cause Though any service-impact operations require twice confirmation in the T2000 NMS, it is likely to be ignored by some maintenance staffs Misoperation of these functions may directly lead your service to abort or the network to out of control Use these operations only when necessary: • Initialize NE data • Full activation • Download • Deactivate a service • Delete a protected subnet • Deactivate a circuit • Loop back (especially loop back the first VC4, which may lead the ECC communication to abort) • Turn off laser • Test laser’s auto turnoff 2005-04-14 Huawei Confidential No disclosure without permission Page 15 of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only • Set overhead byte • Insert an alarm • Reset board • Set clock source (it may cause clock tracking abnormality and hence affect the services) Set VOA Suggestion Double check before performing a dangers operation 3.1.15 Setting Multiplex Section Loop after Downloading an NE Configuration with the NES/RMS For the NES/RMS NMS, pay attention to the following operations: Item Setting multiplex section loop after downloading an NE configuration with the NES/RMS NMS Cause The NES/RMS NMS does not download the multiplex section node parameters after downloading an NE configuration, so the multiplex section switching will fail without resetting the multiplex section loop Suggestion 1.Reset the multiplex section loop after downloading an NE configuration in the NES/RMS NMS Set the multiplex section loop to a direction complying with the primary loop, east-send and west-receive (pay special attention to this item) After loop setting, verify the “I” state of every node 2005-04-14 Huawei Confidential No disclosure without permission Page 16 of 16 ... of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Contents Chapter Purpose Chapter NMS Maintenance Guide Chapter NMS Operation Guide. .. permission 12 Page of 16 T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Keyword: OptiX Optical Transmission NMS Routine Maintenance and Operation Guide Summary: OptiX... T2000 Maintenance and Operation Guide Confidentiality Level: Internal Use Only Χηαπτερ NMS Operation Guide 3.1 NMS Operation Description This section gives some suggestions on the T2000 NMS operation,